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What will you learn from this book? 

Want to learn the Python language without slogging your way through how- 
to manuals? With Head First Python , you’ll quickly grasp Python’s 
fundamentals, working with the built-in data structures and functions. Then 
you’11 move on to building your very own webapp, exploring database man- 
agement, exception handling, and data wrangling. If you’re intrigued by 
what you can do with context managers, decorators, comprehensions, and 
generators, it’s ali here. This second edition is a complete learning experi- 
ence that will help you become a Python programmer in no time. 
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What’s so special about this book? 

Based on the latest research in cognitive Science and learning theory, Head 
First Python uses a visually rich format to engage your mind, rather than a 
text-heavy approach that puts you to sleep. Why waste your time struggling 
with new concepts? This multi-sensory learning experience is designed for 
the way your brain really works. 
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Getting Started Quickly 

Get going with Python programming as quickly as possible. 

In this chapter, we introduce the basies of programming in Python, and we do this in 
typical Head First style: by jumping right in. After just a few pages, you’ll have run 
your first sample program. By the end of the chapter, you’ll not only be able to run the 
sample program, but you’ll understand its code too (and more besides). Along the way, 
you’ll learn about a few of the things that make Python the programming language it is. 
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Working with Data 

AII programs process data, and Python programs are no exception. 

In fact, take a look around: data is everywhere. A lot of, if not most, programming is all about 
data: acquiring data, Processing data, understanding data. To work with data effectively, you need 
somewhere to put your data when Processing it. Python shines in this regard, thanks (in no small 
part) to its inclusion of a handful of widely applicable data structures: lists, dictionaries, tuples, and 
sets. In this chapter, we’ll preview all four, before spending the majority of this chapter digging deeper 
into lists (and we’ll deep-dive into the other three in the next chapter). We’re covering these data 
structures early, as most of what you’ll likely do with Python will revolve around working with data. 
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Working with Structured Data 

PythorTs list data structure is great, but it isn’t a data 

panacea. When you have truly structured data (and using a list to store it may not be 
the best choice), Python comes to your rescue with its built-in dictionary. Out of the box, 
the dictionary lets you store and manipulate any collection of key/value pairs. We look 
long and hard at Python’s dictionary in this chapter, and—along the way—meet set and 
tuple, too. Together with the list (which we met in the previous chapter), the dictionary, 
set, and tuple data structures provide a set of built-in data tools that help to make Python 
and data a powerful combination. 
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Functions and Modules 

Reusing code is key to building a maintainable system. 

And when it comes to reusing code in Python, it all starts and ends with the humble 
function. Take some lines of code, give them a name, and youVe got a function (which 
can be reused). Take a collection of functions and package them as a file, and you’ve 
got a module (which can also be reused). It’s true what they say: it’s good to share, and 
by the end of this chapter, you’ll be well on your way to sharing and reusing your code, 
thanks to an understanding of how PythorVs functions and modules work. 
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bulMing a Webapp 

Getting Real 

At this stage, you know enough Python to be dangerous. 

With this book’s first four chapters behind you, you’re now in a position to productively 
use Python within any number of application areas (even though there’s stili lots of 
Python to leam). Rather than explore the long list of what these application areas 
are, in this and subsequent chapters, we’re going to structure our learning around the 
development of a web-hosted application, which is an area where Python is especially 
strong. Along the way, you’ll learn a bit more about Python. 
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storing and manipulating data 

Where to Put Your Data 

Sooner or later, you’ll need to safely store your data somewhere. 

And when it comes to storing data, Python has you covered. In this chapter, you’11 learn 
about storing and retrieving data from text files, which—as storage mechanisms go—may 
feel a bit simplistic, but is nevertheless used in many problem areas. As well as storing and 
retrieving your data from files, you’ll also learn some tricks of the trade when it comes to 
manipulating data. We’re saving the “serious stuff” (storing data in a database) until the next 
chapter, but there’s plenty to keep us busy for now when working with files. 


Doing Something with Your Webapp’s Data 244 

Python Supports Open, Process, Close 245 

Reading Data from an Existing File 246 

A Better Open, Process, Close: “with” 248 

View the Log Through Your Webapp 254 

Examine the Raw Data with View Source 256 

It 5 s Time to Escape (Your Data) 257 

Viewing the Entire Log in Your Webapp 258 

Logging Specific Web Request Attributes 261 

Log a Single Line of Delimited Data 262 

From Raw Data to Readable Output 265 

Generate Readable Output With HTML 274 

Embed Display Logic in Your Template 275 

Producing Readable Output with Jinj a2 276 

The Current State of Our Webapp Code 278 

Asking Questions of Your Data 279 

Chapter 6’s Code 280 


Form Data 


Vitdb—bikev-0, OettevV, 'aeiouOJ) 


Remote addr 


in.o.o.1 


User_agent 


Results 


Mozilla/5.0 (Madintosb; {V, V} 
Intel M ad 0£ >< IOJIJL) 
Appiciveb^.t/^73^ (mvi/iu 

like £jedko) Cb\rome/^"7 0.Z^Z^> 

.lOt SaWsm* 


xv 






table ofcontents 



using a database 

Putting Python’s DB-API to Use 

Storing data in a relational database system is handy. in this chapter, 

you’ll learn how to write code that interacts with the popular MySQL database technology, using 
a generic database API called DB-API. The DB-API (which comes Standard with every Python 
install) allows you to write code that is easily transferred from one database product to the next... 
assuming your database talks SQL. Although we’ll be using MySQL, there’s nothing stopping you 
from using your DB-API code with your favorite relational database, whatever it may be. Let’s 
see what’s involved in using a relational database with Python. There’s not a lot of new Python in 
this chapter, but using Python to talk to databases is a big deal, so it’s well worth learning. 


Database-Enabling Your Webapp 
Task 1: Install the MySQL Server 
Introducing PythoYs DB-API 

Task 2: Install a MySQL Database Driver for Python 
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a Irtth btt o£ class 

Abstracting Behavior and State 

Classes letyou bundle code behavior and state together. 

In this chapter, you’re setting your webapp aside while you learn about creating Python classes. 
YoiTre doing this in order to get to the point where you can create a context manager with the 
help of a Python class. As creating and using classes is such a useful thing to know about 
anyway, we’re dedicating this chapter to them. We won’t cover everything about classes, but we’ll 
touch on ali the bits you’11 need to understand in order to confidently create the context manager 
your webapp is waiting for. 
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Hooking into Python’s with Statements 

lt’s time to take what yoiTve just learned and put it to work. 

Chapter 7 discussed using a relational database with Python, while Chapter 8 provided an 
introduction to using classes in your Python code. In this chapter, both of these techniques are 
combined to produce a context manager that lets us extend the with statement to work with 
relational database Systems. In this chapter, you’ll hook into the with statement by creating a 
new class, which conforms to Python’s context management protocol. 


What 5 s the Best Way to Share Our Webapp’s Database Code? 336 

Managing Context with Methods 338 

YouVe Already Seen a Context Manager in Aetion 339 

Create a New Context Manager Class 340 

Initialize the Class with the Database Config 341 

Perform Setup with Dunder “enter” 343 

Perform Teardown with Dunder “exit” 345 

Reconsidering Your Webapp Code, 1 of 2 348 

Recalling the “log_request” Function 350 

Amending the c iog_request” Function 351 

Recalling the “view_the_log” Function 352 

It’s Not Just the Code That Changes 353 

Amending the “view_the_log” Function 354 

Answering the Data Questions 359 
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File Edit Window Help Checking our log DB 


$ mysql -u vsearch -p vsearchlogDB 
Enter password: 

Welcome to MySQL monitor... 


mysql> select * from log; 

+ - + - + - + - + - + - + - + 


id 

ts 

phrase | letters 

ip 

browser string 

results | 

1 

2 

3 

4 

2016-03-09 13:40:46 
2016-03-09 13:42:07 
2016-03-09 13:42:15 
2016-03-09 13:43:07 

1--H 

life, the uni ... ything | aeiou 

hitch-hiker j aeiou 

galaxy j xyz 

hitch-hiker j xyz 

1- -+- H 

127.0.0.1 

127.0.0.1 

127.0.0.1 

127.0.0.1 

I--H 

firefox 

safari 

chrome 

firefox 

1--H 

{'u', 'e', 'i', 'a'} 
{'i', 'e'} 

{'y', 'x'} 

set () 

1-- + 


4 rows in set (0.0 sec) 

mysql> quit 
Bye 
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Wrapping Functions 

When it comes to augmenting your code, Chapter 9’s context 
management protocol is not the only game in town. Python aiso iets you 

use function decorators, a technique whereby you can add code to an existing function without 
having to change any of the existing functiorVs code. If you think this sounds like some sort of 
black art, don’t despair: it’s nothing of the sort. However, as coding techniques go, creating a 
function decorator is often considered to be on the harder side by many Python programmers, 
and thus is not used as often as it should be. In this chapter, our plan is to show you that, despite 
being an advanced technique, creating and using your own decorators is not that hard. 


Your Web Server (Not Your Computer) Runs Your Code 

Flask’s Session Technology Adds State 

Dictionary Lookup Retrieves State 

Managing Logins with Sessions 

Let’s Do Logout and Status Checking 

Pass a Function to a Function 

Invoking a Passed Function 
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Processing a List of Arguments 

Accepting a Dictionary of Arguments 

Processing a Dictionary of Arguments 

Accepting Any Number and Type of Function Arguments 

Creating a Function Decorator 

The Final Step: Handling Arguments 

Putting Your Decorator to Work 

Back to Restricting Access to /viewlog 
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What to Do When Things Go Wrong 

Things go wrong, all the time—no matter how good your code is. 

You’ve successfully executed all of the examples in this book, and yoffre likely confident all of 
the code presented thus far works. But does this mean the code is robust? Probably not. Writing 
code based on the assumption that nothing bad ever happens is (at best) naive. At worst, it’s 
dangerous, as unforeseen things do (and will) happen. It’s much better if yoffre wary while 
coding, as opposed to trusting. Care is needed to ensure your code does what you want it to, as 
well as reacts properly when things go south. 


Exception 
+-- 
+ — 
+— 


Stoplteration 

StopAsyncIterat 

ArithmeticError 

A ingPomtError 


_— 

overflowError 

zeroDivisionError 


+ — AssertionError 
+ __ AttributeError 

+ __ BufferError 
+ __ EOFError 


Databases Arefft Always Available 418 
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try Once, but except Many Times 426 
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Learning About Exceptions from “sys” 430 

The Catch-All Exception Handler, Revisited 431 

Getting Back to Our Webapp Code 433 

Silently Handling Exceptions 434 

Handling Other Database Errors 440 

Avoid Tightly Coupled Code 442 

The DBcm Module, Revisited 443 

Creating Custom Exceptions 444 

What Else Can Go Wrong with “DBcm”? 448 

Handling SQLError Is Different 451 

Raising an SQLError 453 

A Quick Recap: Adding Robustness 455 

How to Deal with Wait? It Depends... 456 

Chapter 1 l 5 s Code, 1 of 3 457 
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Dealing with Waiting 

Vour code can sometimes take a long time to execute. 

Depending on who notices, this may or may not be an issue. If some code takes 30 
seconds to do its thing “behind the scenes,” the wait may not be an issue. However, 
if your user is waiting for your application to respond, and it takes 30 seconds, 
everyone notices. What you should do to fix this problem depends on what you’re 
trying to do (and who’s doing the waiting). In this short chapter, we’ll briefly discuss 
some options, then look at one solution to the issue at hand: what happens if 
something takes too long? 


Waiting: What to Do? 462 

How Are You Querying Your Database? 463 

Database INSERTs and SELEGTs Are Different 464 

Doing More Than One Thing at Once 465 

Dofft Get Bummed Out: Use Threads 466 

First Things First: Dofft Panic 470 

Dofft Get Bummed Out: Flash Gan Help 471 

Is Your Webapp Robust Now? 474 

Ghapter 11%’s Code, 1 of 2 475 
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Looping Like Crazy 

lt’s often amazing how much time our programs spend in loops. 

This isn’t a surprise, as most programs exist to perform something quickly a whole heap of times. 
When it comes to optimizing loops, there are two approaches: (1) improve the loop syntax (to 
make it easier to specify a loop), and (2) improve how loops execute (to make them go faster). 
Early in the lifetime of Python 2 (that is, a long, long time ago), the language designers added a 
single language feature that implements both approaches, and it goes by a rather strange name: 
comprehension. 





Reading GSV Data As Lists 479 

Reading GSV Data As Dictionaries 480 

Stripping, Then Splitting, Your Raw Data 482 

Be Gareful When Chaining Method Calis 483 

Transforming Data into the Format You Need 484 

Transforming into a Dictionary Of Lists 485 

Spotting the Pattern with Lists 490 

Gonverting Patterns into Gomprehensions 491 

Take a Closer Look at the Comprehension 492 

Specifying a Dictionary Comprehension 494 

Extend Comprehensions with Filters 495 

Deal with Complexity the Python Way 499 

The Set Comprehension in Aetion 505 

What About “Tuple Gomprehensions”? 507 

Parentheses Around Code == Generator 508 

Using a Listcomp to Process URLs 509 

Using a Generator to Process URLs 510 

Define What Your Function Needs to Do 512 

Yield to the Power of Generator Functions 513 

Tracing Your Generator Function, 1 of 2 514 

One Final Question 518 

Ghapter 12 5 s Gode 519 

It s Time to Go... 520 
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Installing Python 

First thingsfirst: let’s get Python installed on your computer. 

Whether yoiTre running on Windows, Mac OS X, or Linux, Python’s got you covered. How you 
install it on each of these platforms is specific to how things work on each of these operating 
Systems (we know...a shocker, eh?), and the Python community works hard to provide installers 
that target all the popular systems. In this short appendix, you’ll be guided through installing 


Python on your computer. 

Install Python 3 on Windows 522 

Check Python 3 on Windows 523 

Add to Python 3 on Windows 524 

Install Python 3 on Mac OS X (macOS) 525 

Check and Configure Python 3 on Mac OS X 526 

Install Python 3 on Linux 527 


pydi°nctny\vWe 

Deploying Your Webapp 

At the end of Chapter 5, we claimed that deploying your webapp to 
the cloud was only 10 minutes away. It’s now time to make good on that promise. 
In this appendix, we are going to take you through the process of deploying your webapp on 
PythonAnywhere, going from zero to deployed in about 10 minutes. PythonAnywhere is a 
favorite among the Python programming community, and it’s not hard to see why: it works exactly 
as you’d expect it to, has great support for Python (and Flask), and—best of all—you can get 


started hosting your webapp at no cost. 

Step 0: A Little Prep 530 

Step 1: Sign Up for PythonAnywhere 531 

Step 2: Upload Your Files to the Cloud 532 

Step 3: Extract and Install Your Code 533 

Step 4: Create a Starter Webapp, 1 of 2 534 

Step 5: Configure Your Webapp 536 

Step 6: Take Your Cloud-Based Webapp for a Spin! 537 
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There’s Always More to Learn 

It was never our intentiori to try to cover everything. tns books goai was 

always to show you enough Python to get you up to speed as quickly as possible. There’s a lot 
more we could’ve covered, but didn’t. In this appendix, we discuss the top 10 things that—given 
another 600 pages or so—we would’ve eventually gotten around to. Not all of the 10 things will 
interest you, but quickly flip through them just in case we’ve hit on your sweet spot, or provided 
an answer to that nagging question. AII the programming technologies in this appendix come 
baked in to Python and its interpreter. 


1. What About Python 2? 540 

2. Virtual Programming Environments 541 

3. More on Object Orientation 542 

4. Formats for Strings and the Like 543 

5. Getting Things Sorted 544 

6. More from the Standard Library 545 

7. Running Your Gode Concurrently 546 

8. GUIs with Tkinter (and Fun with Turtles) 547 

9. It s Not Over 5 Til It’s Tested 548 

10. Debug, Debug, Debug 549 
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top ten projects ri9t coVered 

Even More Tools, Libraries, and Modules 

We know what you’re thinking as you read this appendix’s title. 

Why on Earth didn’t they make the title of the last appendix: The Top Twenty Things We Didrit 
CoverT Why another 10? In the last appendix, we limited our discussion to stuff that comes 
baked in to Python (part of the language’s “batteries included”). In this appendix, we cast the net 
much further afield, discussing a whole host of technologies that are available to you because 
Python exists. There’s lots of good stuff here and—just like with the last appendix—a quick 
perusal won’t hurt you one single bit. 


1. Alternatives to >>> 

552 

2. Alternatives to IDLE 

553 

3. Jupyter Notebook: The Web-Based IDE 

554 

4. Doing Data Science 

555 

5. Web Development Technologies 

556 

6. Working with Web Data 

557 

7. More Data Sources 

558 

8. Programming Tools 

559 

9. Kivy: Our Pick for “Coolest Project Ever” 

560 

10. Alter native Implementations 

561 
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The Python Community 

Python is much more than a great programming language. 

Ifs a great community, too. The Python Community is welcoming, diverse, open, friendly, 
sharing, and giving. We’re just amazed that no one, to date, has thought to put that on a 
greeting card! Seriously, though, there’s more to programming in Python than the language. An 
entire ecosystem has grown up around Python, in the form of excellent books, blogs, websites, 
conferences, meetups, user groups, and personalities. In this appendix, we take a survey of the 
Python community and see what it has to offer. Don’t just sit around programming on your own: 
get involved! 
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Which Book Should I Read Next? 568 
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how to use this book 


Who Is This Pook For? 

If you can answer “yes” to all of these: 


o 

o 

o 


Do you already know how to program in another 
programming language? 

Do you wish you had the know-how to program Python, 
add it to your list of tools, and make it do new things? 

Do you prefer actually doing things and applying the stuff 
you learn over listening to someone in a lecture rattle on 
for hours on end? 


this book is for you. 


Who should probably back away front this book? 

If you can answer “yes” to any of these: 


o 

o 

o 


Do you already know most of what you need to know to 
program with Python? 

Are you looking for a reference book to Python, one that 
covers all the details in excruciating detail? 

Would you rather have your toenails pulled out by 15 
screaming monkeys than learn something new? Do you 
believe a Python book should cover everything and if it 
bores the reader to tears in the process, then so much 
the better? 


Tltis is NOT a 

reference fook, 
and we assume 
you Ve programm 
tefore. 


this book is not for you. 


f 


■=? 
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' s , wi 'tb a dvedi-fc dav-d 
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the intro 


We Kwow What YouYe Thiwkmg 


u 


u 


How can this be a serious Python book? 
Whafs with all the graphics?” 

Can I actually leam it this way?” 


?5 


& ihi»ks 


m ^ i 


We kwow what your brain is thiwkmg 

Your brain craves novelty. It 5 s always searching, scanning, waiting for something 
unusual. It was built that way, and it helps you stay alive. 

So what does your brain do with all the routine, ordinary, normal things 
you encounter? Everything it can to stop them from interfering with the 
brain’s realjob —recording things that matter. It doesn’t bother saving the 
boring things; they never make it past the “this is obviously not important” 
filter. 

How does your brain know what’s important? Suppose you’re out for a day 
hike and a tiger jumps in front of you, what happens inside your head and 
body? 

Neurons fire. Emotions crank up. Chemicals surge. 

And that 5 s how your brain knows... 

This must be important! Don’t forget it! 

But imagine you’re at horne, or in a library. It’s a safe, warm, tiger-free zone. ir '"‘a -r* 
You re studymg. Gettmg ready for an exam. Or trymg to learn some tough 
technical topic your boss thinks will take a week, 10 days at the most. 

Just one problem. Your brain’s trying to do you a big favor. It’s trying 
make sure that this obviously nonimportant content doesn’t clutter up 
resources. Resources that are better spent storing the really big things 
Like tigers. Like the danger of fire. Like how you should never have 
posted those “party” photos on your Facebook page. And there 5 s no 
simple way to teli your brain, “Hey brain, thank you very much, but 
no matter how dull this book is, and how little I’m registering on the 
emotional Richter scale right now, I really do want you to keep this 
stuff around.” 
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Metacognition: Thinking About Thinking 

If you really want to learn, and you want to learn more quickly and more 
deeply, pay attention to how you pay attention. Think about how you think. 
Learn how you learn. 

Most of us did not take courses on metacognition or learning theory when we 
were growing up. We were expected to learn, but rarely taught to learn. 




The trick is to get your brain to see the new material you’re learning as 
Really Important. Grucial to your well-being. As important as a tiger. 
Otherwise, you’re in for a constant battle, with your brain doing its best to 
keep the new content from sticking. 


But we assume that if you’re holding this book, you really want to learn how 
to solve programming problems with Python. And you probably don’t want to 
spend a lot of time. If you want to use what you read in this book, you need to 
remember what you read. And for that, you Ve got to understand it. To get the most 
from this book, or any book or learning experience, take responsibility for your 
brain. Your brain on this content. 


So just how DO you get your brain to treat 
programming like it was a hungry tiger? 


There’s the slow, tedious way, or the faster, more effective way. The 
slow way is about sheer repetition. You obviously know that you are able to learn 
and remember even the dullest of topics if you keep pounding the same thing into your 
brain. With enough repetition, your brain says, “This doesn’t feel important to him, but he 
keeps looking at the same thing over and over and over , so I suppose it must be.” 


The faster way is to do any thing that increases brain activity, especially different 
types of brain activity. The things on the previous page are a big part of the solution, 
and theyVe all things that have been proven to help your brain work in your favor. For 
example, studies show that putting words within the pictures they describe (as opposed to 
somewhere else in the page, like a caption or in the body text) causes your brain to try to 
makes sense of how the words and picture relate, and this causes more neurons to fire. 
More neurons firing = more chances for your brain to get that this is something worth 
paying attention to, and possibly recording. 


A conversational style helps because people tend to pay more attention when they 
perceive that theyVe in a conversation, since theyVe expected to follow along and hold up 
their end. The amazing thing is, your brain doesn’t necessarily care that the “conversation” 
is between you and a book! On the other hand, if the writing style is formal and dry, your 
brain perceives it the same way you experience being lectured to while sitting in a roomful 
of passive attendees. No need to stay awake. 


But pictures and conversational style are just the beginning... 
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Here'$ What WE Pid: 

We used pictures, because your brain is tuned for visuals, not text. As far as your brairfs 
concerned, a picture really is worth a thousand words. And when text and pictures work 
together, we embedded the text in the pictures because your brain works more effectively 
when the text is within the thing the text refers to, as opposed to in a caption or buried in the 
text somewhere. 

We used redundancy , saying the same thing in different ways and with different media types, 
and multiple senses , to increase the chance that the content gets coded into more than one area 
of your brain. 

We used concepts and pictures in unexpected ways because your brain is tuned for novelty, 
and we used pictures and ideas with at least some emotional content , because your brain 
is tuned to pay attention to the biochemistry of emotions. That which causes you to feel 
something is more likely to be remembered, even if that feeling is nothing more than a little 

humor , surprise , or interest. 

We used a personalized, conversational style , because your brain is tuned to pay more 
attention when it believes you’re in a conversation than if it thinks you’re passively listening 
to a presentation. Your brain does this even when you’re reading. 

We included more than 80 activities , because your brain is tuned to learn and remember 
more when you do things than when you read about things. And we made the exercises 
challenging-yet-doable, because that’s what most people prefer. 

We used multiple learning styles , because you might prefer step-by-step procedures, while 
someone else wants to understand the big picture first, and someone else just wants to see 
an example. But regardless of your own learning preference, everyone benefits from seeing the 
same content represented in multiple ways. 

We include content for both sides of your brain , because the more of your brain you 
engage, the more likely you are to learn and remember, and the longer you can stay focused. 
Since working one side of the brain often means giving the other side a chance to rest, you 
can be more productive at learning for a longer period of time. 

And we included stories and exercises that present more than one point of viezv, 
because your brain is tuned to learn more deeply when it’s forced to make evaluations and 
judgments. 

We included challenges, with exercises, and asked questions that don’t always have a 
straight answer, because your brain is tuned to learn and remember when it has to work at 
something. Think about it—you can’t get your body in shape just by watching people at the 
gym. But we did our best to make sure that when you’re working hard, it’s on the right things. 
That you We not spending one extra dendrite processing a hard-to-understand example, 
or parsing difficult, jargon-laden, or overly terse text. 

We used people. In stories, examples, pictures, and so on, because, well ,youPe a person. And 
your brain pays more attention to people than it does to things. 
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Here'$ what YOU can do to bend 
your brain into submissiori 

So, we did our part. The rest is up to you. These tips are a 
starting point; listen to your brain and figure out what works 
for you and what doesn’t. Try new things. 

Cu-fc this out dhd stidk it 

otn youir ire-fv-igei-a-tov. 


O Slow down. The more you understand, the 
less you have to memorize. 

Don’t just read. Stop and think. When the book asks 
you a question, don’t just skip to the answer. Imagine 
that someone really is asking the question. The 
more deeply you force your brain to think, the better 
chance you have of learning and remembering. 


0 Do the exercises. Write your own notes. 

We put them in, but if we did them for you, that 
would be like having someone else do your workouts 
for you. And don’t just look at the exercises. Use a 
pencil. There’s plenty of evidence that physical 
activity while learning can increase the learning. 


o Read the “There Are No Dumb Questions” 
sections. 

That means all of them. They’re not optional 
sidebars, they We part of the core content! 

Don’t skip them. 

0 Make this the last thing you read before bed. 
Or at least the last challenging thing. 

Part of the learning (especially the transfer to 
long-term memory) happens after you put the book 
down. Your brain needs time on its own, to do more 
processing. If you put in something new during that 
Processing time, some of what you just learned will 
be lost. 


(D T,lk abou.it. Out lo„d. 

Speaking activates a different part of the brain. If 
you’re trying to understand something or increase 
your chance of remembering it later, say it out loud. 
Better stili, try to explain it out loud to someone else. 
You’11 learn more quickly, and you might uncover 
ideas you hadn’t known were there when you were 
reading about it. 


o Drink water. Lots of it. 

Your brain works best in a nice bath of fluid. 
Dehydration (which can happen before you ever 
feel thirsty) decreases cognitive function. 

o Listen to your brain. 

Pay attention to whether your brain is getting 
overloaded. If you find yourself starting to skim 
the surface or forget what you just read, it 5 s time 
for a break. Once you go past a certain point, you 
won’t learn faster by trying to shove more in, and 
you might even hurt the process. 

o Feel something. 

Your brain needs to know that this matters. Get 
involved with the stories. Make up your own 
captions for the photos. Groaning over a bad joke 
is stili better than feeling nothing at all. 

0 Write a lot of code! 

There’s only one way to learn to program in Python: 
write a lot of code. And that 5 s what you’re going 
to do throughout this book. Goding is a skill, and 
the only way to get good at it is to practice. We’re 
going to give you a lot of practice: every chapter has 
exercises that pose a problem for you to solve. Don’t 
just skip over them—a lot of the learning happens 
when you solve the exercises. We included a solution 
to each exercise—don’t be afraid to peek at the 
solution if you get stuck! (It’s easy to get snagged 
on something small.) But try to solve the problem 
before you look at the solution. And definitely get it 
working before you move on to the next part of the 
book. 
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Read Me, 1 of 2 

This is a learning experience, not a reference book. We deliberately stripped out everything that 
might get in the way of learning whatever it is we’re working on at that point in the book. And 
the first time through, you need to begin at the beginning, because the book makes assumptions 
about what you Ve already seen and learned. 

This book is designed to get you up to speed as quickly as possible. 

As you need to know stuff, we teach it. So you won’t find long lists of technical material, no 
tables of Python’s operators, nor its operator precedence rules. We don’t cover everything , but 
weVe worked really hard to cover the essential material as well as we can, so that you can get 
Python into your brain quickly and have it stay there. The only assumption we make is that you 
already know how to program in some other programming language. 

This book targets Python 3 

We use Release 3 of the Python programming language in this book, and we cover how to get 
and install Python 3 in Appendix A. This book does not use Python 2. 


We put Python to work for you right away. 

We get you doing useful stuff in Ghapter 1 and build from there. There’s no hanging around, 
because we want you to be productive with Python right away. 

The activities are NOT optional—you have to do the work. 

The exercises and activities are not add-ons; theyVe part of the core content of the book. Some 
of them are to help with memory, some are for understanding, and some will help you apply 
what you Ve learned. Don y t skip the exercises. 

The redundancy is intentional and important. 

One distinet difference in a Head First book is that we want you to really get it. And we want 
you to finish the book remembering what you Ve learned. Most reference books don’t have 
retention and recall as a goal, but this book is about learning , so you’11 see some of the same 
concepts come up more than once. 

The examples are as lean as possible. 

Our readers teli us that it’s frustrating to wade through 200 lines of an example looking for the 
two lines they need to understand. Most examples in this book are shown within the smallest 
possible context, so that the part youVe trying to learn is ciear and simple. Don’t expect all of 
the examples to be robust, or even complete—they are written specifically for learning, and 
aren’t always fully functional (although weVe tried to ensure as much as possible that they are). 
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Read Me, 2 of 2 

Yes, there’s more... 

This second edition is NOT at all like the first. 

This is an update to the first edition of Head First Python , which published late in 2010. 
Although that book and this one share the same author, he’s now older and (hopefully) 
wiser, and thus, decided to completely rewrite the first editiorfs content for this edition. 
So ...everything is new: the order is different, the content has been updated, the examples 
are better, and the stories are either gone or have been replaced. We kept the cover— 
with minor amendments—as we figured we didn’t want to rock the boat too much. It’s 
been a long six years...we hope you enjoy what weVe come up with. 

Where’s the code? 

WeVe placed the code examples on the Web so you can copy and paste them as needed 
(although we do recommend that you type in the code as you follow along). You’11 find the 
code at these locations: 


http://bit . ly/head-first-python-2e 
http://python. itcarlozv . ie 
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1 th& bcfcsics 


^ Getting Started Quickly + 



Somebodys obviously spent 
far too many days at sea... 


Wha+'s Python? A nonvenomous 
snake? A late 1960s comedy troupe? 
A programming language? Gosh! It's 
ali of these things! 


Get going with Python programming as quickly as possible. 

In this chapter, we introduce the basies of programming in Python, and we do this in 
typical Head First style: by jumping right in. After just a few pages, you’ll have run your 
first sample program. By the end of the chapter, you’11 not only be able to run the sample 
program, but you’ll understand its code too (and more besides). Along the way, you’ll learn 
about a few of the things that make Python the programming language it is. So, let’s not 
waste any more time. Flip the page and let’s get going! 


this is a new chapter 









say hello — not! 


Preakiwg with Traditiow 


Pick up almost any book on a programming language, and the first thing 
you’11 see is the Hello World example. 


0 


O 




No, we aren’t. 

This is a Head First book, and we do things 
differently Vound here. With other books, 
there is a tradition to start by showing you 
how to write the Hello World program in the 
language under consideration. However, 
with Python, what you end up with is a 
single statement that invokes Python’s 
built-in print function, which displays 
the traditional “Hello, World!” message 
on screen. It’s almost too exciting...and it 
teaches you next to nothing. 

So, no, we aren’t going to show you the Hello 
World program in Python, as there’s really 
nothing to learn from it. We’re going to take 
a different path... 


Start i ng with a meatier example 

Our plan for this chapter is to start with an example that’s somewhat larger 
and, consequently, more useful than Hello World. 

We’ll be right up front and teli you that the example we have is somewhat 
contrived : it does do something, but may not be entirely useful in the long run. 
That said, weVe chosen it to provide a vehicle with which to cover a lot of 
Python in as short a timespan as possible. And we promise by the time you Ve 
worked through the first example program, you’11 know enough to write Hello 
World in Python without our help. 
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Juwp Right In 

If you haven’t already installed a version of Python 3 on your computer, 
pause now and head on over to Appendix A for some step-by-step installation 
instructions (it’ll only take a couple minutes, promise). 

With the latest Python 3 installed, you’re ready to start programming 
Python, and to help with this—for now—we’re going to use Python’s built-in 
integrated development environment (IDE). 


Python'$ IPIE is all you need to get going 


When you install Python 3 on your computer, you also get a very simple yet 
usable IDE called IDLE. Although there are many different ways in which to 
run Python code (and youdl meet a lot of them throughout this book), IDLE 
is all you need when starting out. 

Start IDLE on your computer, then use the File... ~>JVew File... menu option to 
open a new editing window. When we did this on our computer, we ended up 
with two windows: one called the Python Shell and another called Untitled: 


Python 3.4.3 Shell 


TWis v/mdov/ ?°? s 

i\rs*t* TWmk Jc 
rt as -the u *fiv*st 
y/mdow- 


1 


pF* y* *itti 
Ne* File..., 

« nk 

Se£ °»d wi.dow." 


Python 3.4.3 (v3.4.3:9b73flc3e601, Feb 23 2015, 02:52:03) 

[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin 

Type "Copyright”, "credits 11 or "license()" for more information 

»> 

# O • Untitled 




|PL-6) “then 

thoo"Pile-• _+ Ke'M 
ple..." vesults '*> two - 
vumdows a^eavm0| on 




sfiree*. 



you are here ► 
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Iet’s get going 


Uwderstawdmg IPLE'$ Windows 

Both of these IDLE windows are important. 

The first window, the Python Shell, is a REPL environment used to run 
snippets of Python code, typically a single statement at a time. The more 
you work with Python, the more you’11 come to love the Python Shell, 
and you’11 be using it a lot as you progress through this book. For now, 
though, we are more interested in the second window. 

The second window, Untitled, is a text editing window that can be used 
to write complete Python programs. It’s not the greatest editor in the 
world (as that honor goes to <insertyourfavorite text editor’s name here >), but 
IDLE’s editor is quite usable, and has a bunch of modern features built 
right in, including color-syntax handling and the like. 

As we are jumping right in, let’s go ahead and enter a small Python 
program into this window. When you are done typing in the code below, 
use the File...—>Save... menu option to save your program under the name 
odd. py. 

Be sure to enter the code exactly as shown here: 



What does REPL mean? 

It's geek shorthand for"read- 
eval-print-loop,"and describes an 
interactive programming tool that 
lets you experiment with snippets of 
code to your hearfs desire. Find out 
way more than you need to know by 
visiting http://en. wikipedio.org/wiki/ 
Read-evai-printjoop. 


pon’t wovvy abou-t 
wha*t this dode 
does -fov now- v)us*b 
-type i-t in*fco “tbc 
edrtin^ window- 

Be suve bo save ft 
as w oddyy” beW 

dontmuincy 



# O # odd.py - /Users/Paul/Desktop/_Ne-wBook/ch01/odd.py {3.4.3) 


from datetime iraport datetime 


odds = [ 1, 

3, 5, 

7, 

9, 

11, 

13, 15, 

17, 

19, 

21, 

23, 25, 

27, 

29, 

31, 

33, 35, 

37, 

39, 

41, 

43, 45, 

47, 

49, 

51, 

53, 55, 

57, 

59 ] 


right_this_minute = datetime.today().minute 

if right_this_minute in odds: 

print( r This minute seems a little odd.) 
elsei 

print( Not an odd minute. 1 ) 


Ln: 15 


Coi: 0 


&o...now what? If you’re anything like us, you can’t wait to run this code, right? 
Let’s do this now. With your code in the edit window (as shown above), press the 
F5 key on your keyboard. A number of things can happen... 
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What Happews Next... 

If your code ran without error, flip over to the next page, and keep going. 

If you forgot to save your code before you tried to run it, IDLE complains, as 
you have to save any new code to a file first. You’11 see a message similar to 
this one if you didn’t save your code: 



Source Must Be Saved 
OK to Save? 


Cancel 


OK 


By de-Pault, IDLB 

vun £ode that 
hash t been saved- 



Glick the OK button, then provide a name for your file. WeVe chosen odd 
as the name for our file, and weVe added a . py extension (which is a Python 
convention well worth adhering to): 




You ave -Pvee to use 
whatevev name you 
like tov youv f^va», 

but it’s \>vobably 
best—'t you v* 

alo»5—to 

stidk -to tbe same 


If your code now runs (having been saved), flip over to the next page, and keep 
going. If, however, you have a syntax error somewhere in your code, you’11 see 
this message: 

invalid syntax 


OK 



Glick the OK button, then note where IDLE thinks the syntax error is: look 
for the large red block in the edit window. Make sure your code matehes ours 
exactly, save your file again, and then press F5 to ask IDLE to exeeute your 
code once more. 


name 3s vs. 


As you ean no doubt 
teli, IVL-B isn t gveat 
stating y/hai the 
syntax evvov is. But 
dliek 0^, and a lav^e 
ved bloek indieates 
wheve IVLB thinks 
the pvoblem is. 
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pressing F5 works! 


Press F 5 to Ruw Your Code 

Pressing F5 exeeutes the code in the currendy selected IDLE text-editing 
window—assuming, of course, that your code doesn’t contain a runtime error. 
If you have a runtime error, you’11 see a Traceback error message (in red). 
Read the message, then return to the edit window to make sure the code you 
entered is exactly the same as ours. Save your amended code, then press F5 
again. When we pressed F5, the Python Shell became the active window, and 
here 5 s what we saw: 



u 


Fvorw -fehis poih-fc oh; 
we'll re-fev io "ih e 

IVLB text—editing 

wihdow^ simply 
edrfc wihdow” 


Python 3.4.3 Shell 


Python 3.4.3 (v3.4.3:9b73fIc3e601, Feb 23 2015, 02:52:03) 

[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin 

Type "Copyright", "credits" or "license()" for more information. 

»> ================================ RESTART =================== 

po**t Yiovvy i-f you sec a 

w»essay. Read 

^ leav* why *tWs is. 


This minute seems a little odd. 
»> 


OY\ 


Ln: 7 Coi: 4 


Depending on what time of day it is, you may have seen the JVot an odd minute 
message instead. Don’t worry if you did, as this program displays one or the 
other message depending on whether your computeFs current time contains 
a minute value that’s an odd number (we did say this example was contrived , 
didn’t we?). If you wait a minute, then click the edit window to select it, then 
press F5 again, your code runs again. Youdl see the other message this time 
(assuming you waited the required minute). Feel free to run this code as often 
as you like. Here is what we saw when we (very patiently) waited the required 
minute: 



pvessihO) F5 */Wile m the 
edit WmdoxM vuhS youv 
tode, theh displays the 

v-esultmo, output m the 

Pythoh Shell- 


# # # Python 3.4.3 Shell 

Python 3.4.3 (v3.4.3:9b73fIc3e601, Feb 23 2015, 02:52:03) 

[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin 

Type "Copyright", "credits" or "license()" for more information. 

»> ================================ RESTART =================== 

»> 

This minute seems a little odd. 

»> ================================ RESTART =================== 

»> 

Not an odd minute. 

»> 


Ln: 10 Coi: 4 


Let’s spend some time learning how this code runs. 
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Code Runs Iwwediately 


When IDLE asks Python to run the code in the edit window, Python starts at 
the top of the file and begins exeeuting code straightaway. 


For those of you coming to Python from one of the G-like languages, note 
that there is no notion of a main () function or method in Python. There’s 
also no notion of the familiar edit-compile-link-run process. With Python, 
you edit your code and save it, and run it immediately. _ 


O 


Hang on a second. You said "IDLE asks 
Python to run the code"...but isn't Python the 
programming language and IDLE the IDE? If so, 
whafs actually doing the running here?!? 



Oh, good catch. That is confusing. 

Here’s what you need to know: “Python” is the name given to the 
programming language and “IDLE” is the name given to the built-in 
Python IDE. 

That said, when you install Python 3 on your computer, an interpreter 
is installed, too. This is the technology that runs your Python code. Rather 
confusingly, this interpreter is also known by the name “Python.” By 
right, everyone should use the more correct name when referring to this 
technology, which is to call it “the Python interpreter.” But, alas, nobody 
ever does. 

Starting this very second, in this book, we’ll use the word “Python” 
to refer to the language, and the word “interpreter” to refer to the 
technology that runs your Python code. “IDLE” refers to the IDE, which 
takes your Python code and runs it through the interpreter. It 5 s the 
interpreter that does all the actual work here. 


tfiereiqre no 

Dumb Questions 



Is the Python interpreter something like the Java VM? 


Yes and no. Yes, in that the interpreter runs your code. But 
no, in how it does it. In Python, there’s no real notion of your source 
code being compiled into an “executable.” Unlike the Java VM, the 
interpreter doesn’t run . class files, it just runs your code. 



But, surely, compilation has to happen at some stage? 


Yes, it does, but the interpreter does not expose this process 
to the Python programmer (you). AII of the details are taken care of 
for you. AII you see is your code running as IDLE does all the heavy 
lifting, interacting with the interpreter on your behalf. We’ll talk more 
about this process as this book progresses. 


you are here ► 
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Executing Code, One Statewent at a Time 


Here is the program code from page 4 again: 


from datetime import datetime 


[ 1 , 

3 , 

5 , 

7 , 

9 , 

11 , 

13 , 

15 , 

17 , 

19 , 

21 , 

23 , 

25 , 

27 , 

29 , 

31 , 

33 , 

35 , 

37 , 

39 , 

41 , 

43 , 

45 , 

47 , 

49 , 

51 , 

53 , 

55 , 

57 , 

59 ] 


right_this_minute = datetime.today().minute 

if right_this_minute in odds: 

print( M This minute seems a little odd. M ) 
else: 

print( M Not an odd minute.") 


let's be the Python interpreter 

Let 5 s take some time to run through this code in much the same way that the 
interpreter does, line by line, from the top of the file to the bottom. 

The first line of code imports some preexisting functionality from Python’s 
Standard library, which is a large stock of Software modules providing lots 
of prebuilt (and high-quality) reusable code. 


Ttiink of modules 
as a collection of 
related functions. 


In our code, we specifically request one submodule from the Standard 
library’s datetime module. The fact that the submodule is also called 
datetime is confusing, but that’s how this works. The datetime 
submodule provides a mechanism to work out the time, as you’11 see over the 
next few pages. 


This is -the *a.»e ot 
■the s-ta^daircl iibv-av-y 
"«xMe b> ir«po*-t -the 
reusable Code -Grom. 


from datetime import datetime 

J?\ 


-r- 


This is -the 
ndmC the 
submodule* 


[ 1 , 

3 , 

5 , 

7 , 

9 , 

11 , 

13 , 

15 , 

17 , 

19 , 

21 , 

23 , 

25 , 

27 , 

29 , 

31 , 

33 , 

35 , 

37 , 

39 , 

41 , 

43 , 

45 , 

47 , 

49 , 

51 , 

53 , 

55 , 

57 , 

59 ] 


| n ibis book, whe* we ^«t you to ^Y jaUitu 1 ^ 
attentio* io a Ime ot Code, we b^bl^bi .i <yst 
like we did here)- 


the 
starts 
* tte fe, ot the 

tilc 3 \r\d Works dowh 
Wd the botw 
exeeutmg eadh Ime 
oi Python dode as 
•t goes. 
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Functions + Modules * The Standard Library 

Python’s Standard library is very rich , and provides a lot of reusable code. 

Let’s look at another module, called os, which provides a platform-independent way 
to interact with your underlying operating system (wedl return to the date time 
module in a moment). Let’s concentrate on just one provided function, getcwd, 
which—when invoked—returns your current working directory. 

Here’s how you’d typically import , then invoke , this function within a Python program: 


|r*yov“t “the ‘funfrbo* 

-fvom vis m odule- 



from os import getcwd 


where am I = getcwd() 


•••■fcheK ‘mvoke as 
ve^uired. 


A collection of related functions makes up a module, and there are lots of 
modules in the Standard library: 


The •function— 




getcwd 


...is yarl o-f a 
module 


...whith tomes as Y >a ' r "t 
■the standavd libv-av-y 


Functions are inside 
modules inside the 
Standard lihrary. 



enum 


random 


datetime 


t>od wwrry about what eath of these adules does at this staoie. 
Wc have a qwtk pveviev, of some of them ovev the page, and will 
see move o+ the vest latev in this book. 
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digging deeper 



TJp Cl«se wifl the Standard .Library 


The Standard library is the jewel in Python’s crown, supplying reusable modules that help you with 
everything from, for example, working with data, through manipulating ZIP archives, to sending emails, 
to working with HTML. The Standard library even includes a web server, as well as the popular SQLite 
database technology. In this Up Cio se, well present an overview of just a few of the most commonly used 
modules in the Standard library To follow along, you can enter these examples as shown at your >>> 
prompt (in IDLE). If you are currently looking at IDLE’s edit window, choose Run...—>Python Shell from the 
menu to access the >>> prompt. 

Let’s start by learning a little about the System your interpreter is running on. Although Python prides 
itself on being cross-platform, in that code written on one platform can be executed (generally unaltered) 
on another, there are times when it’s important to know that you are running on, say, Mac OS X. The sys 
module exists to help you learn more about your interpreter 5 s System. Here’s how to determine the identity 
of your underlying operating system, by first importing the sys module, then accessing the platform 
attribute: 


»> import sys 
»> sys.platform 

1 darwin' 



Upoiri -the ">odwle you »eed, the» addess the attribute 
o+ mterest- It looks like we are *-u»»i»a "daru/m", 
whidh is the Mad OS. )< ker»el »ame. 


The sys module is a good example of a reusable module that primarily provides access to preset attributes 
(such as platform). As another example, here’s how to determine which version of Python is running, 
which we pass to the print function to display on screen: 

There's a lot ot m-formatio» about 
the Pytho» versio» weVe ru»»i»<), 


»> print (sys .version) 

3.4.3 (v3.4.3:9b73fIc3e601, Feb 23 2015, 02:52:03) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] 


mdudm$ -bba-b iVs 


The os module is a good example of a reusable module that primarily yields functionality, as well as 
providing a system-independent way for your Python code to interact with the underlying operating system, 
regardless of exactly which operating system that is. 

For example, here 5 s how to work out the name of the folder your code is operating within using the 
getcwd function. As with any module, you begin by importing the module before invoking the function: 


»> import os 
»> os . getcwd () 

'/Users/HeadFirst/CodeExamples' 


} 


_ l»>port the module, the» i»voke the 
-Fu»dtio»ality you »eed- 


You can access your systenfs environment variables, as a whole (using the environ attribute) or 
individually (using the getenv function): 

»> os. environ 

'environ({'XPC_FLAGS': '0x0', 'HOME': '/Users/HeadFirst', 'TMPDIR' 

folders/18/t93gmhc546b7b2cngfhzl0100000gn/T /', ... 'PYTHONPATH': 

Python 3.4/IDLE.app/Contents/Resources', ... 'SHELL': '/bin/bash', 

'flATgetlvCHOME') ^ 3 ^med 

'/Users/HeadFirst' -- dEfHbuEe (Tlrom “tbc d«l*kc) £oirtchKed in 

1 ChVi\roh W ) usmg "(jefcehv" 


' Chvivoh 

d-fc-tmibu-fce 

^Oh*t^ihS lo*ts 

' /var/ 1 da^ta* 

/Applications /f 
'USER' : 
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XJp C]«se wlfb iR ^anJarJ Library, Confirmet! <X 


Working with dates (and times) comes up a lot, and the Standard library provides the date time module to 
help when you’re working with this type of data. The date . today function provides today 5 s date: 


»> import datetime 
»> datetime. date. today () 
datetime.date(2015, 5, 31) 


Today s date 


That’s certainly a strange way to display today’s date, though, isn’t it? You can access the day, month, and 
year values separately by appending an attribute access onto the call to date . today: 


»> datetime. date. today () . day 

31 

»> datetime. date. today () . month 

5 

»> datetime. date. today () . year 

2015 


The Component parts o-P 
iodays date 


You can also invoke the date . isof ormat function and pass in today 5 s date to display a much more user- 
friendly version of today 5 s date, which is converted to a string by isof ormat: 

»> datetime. date. isof ormat (datetime. date. today () ) foday s date as 3 sfTm^ 

'2015-05-31' 

And then there’s time, which none of us seem to have enough of. Can the Standard library teli us what time it 
is? Yes. After importing the time module, call the strf time function and specify how you want the time 
displayed. In this case, we are interested in the current time’s hours (%H) and minutes (%M) values in 24-hour 
format: 

»> import time 
»> time, strf time ("%H: %M") 

'23:55' «=_ 


<$ood heavens! Is that the tii*e« 


;p 


How about working out the day of the week, and whether or not it’s before noon? Using the %A 
specification with strf time does just that: 

»> time. strf time (" %A %p") Wfcve novi \worked owi that it's tive mmutes io midnijK-t 
Sunday PM --' on Sunday -fov- bed, ferhaps? 

As a final example of the type of reusable functionality the Standard library provides, imagine you have some 
HTML that you are worried might contain some potentially dangerous <script> tags. Rather than 
parsing the HTML to detect and remove the tags, why not encode all those troublesome angle brackets 
using the e scape function from the html module? Or maybe you have some encoded HTML that you’d 
like to return to its original form? The une scape function can do that. Here are examples of both: 


»> import html 

»> html.escape("This HTML fragment contains a <script>script</script> tag.") 
'This HTML fragment contains a &lt;script&gt;script&lt;/script&gt; tag.' 

»> html.unescape("I fihearts; Python's &lt;Standard library&gt;.") 

"I V Python's <standard library>." ^ 


Cohvcv-fcihg 
. “to dhd 

-(Vor* HT/l/JL 
ty\todtd tex-1 
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everything you need 


Patteries Included 



Yes. That’s what they mean. 

As the Standard library is so rich, the thinking is all 

you need to be immediately productive with 
the language is to have Python installed. 

Unlike Ghristmas morning, when you open 
your new toy only to discover that it doesn’t 
come with batteries, Python doesn’t disappoint; 
it comes with everything you need to get going. 
And it 5 s not just the modules in the Standard 
library that this thinking applies to: don’t forget 
the inclusion of IDLE, which provides a small, 
yet usable, IDE right out of the box. 

All you have to do is code. 


tfieretqre no 

Dumb Questions 



Q: How am I supposed to work out what any particular 
module from the Standard library does? 


.The Python documentation has all the answers on 
the Standard library. Here’s the kicking-off point: https://docs. 
python, org/3/library/index.html. 


The Standard library isn't the only place you'11 
find excellent importable modules to use with 
your code. The Python community also supports a 
thriving collection of third-party modules, some of 
which we'll explore later in this book. If you want a 
preview, check out the community-run repository: 
h ftp://pypi.pyth on.org. 
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Pata Structores Come Puilt-in 


As well as coming with a top-notch Standard library , Python also has some 
powerful built-in data structures. One of these is the list, which can be 
thought of as a very powerful array. Like arrays in many other languages, lists 
in Python are enclosed within square brackets ([ ]). 


The next three lines of code in our program (shown below) assign a literal 
list of odd numbers to a variable called odds. In this code, odds is a list of 
integers , but lists in Python can contain any data of any type, and you can even 
mix the types of data in a list (if that 5 s what you’re into). Note how the odds 
list extends over three lines, despite being a single statement. This is OK, as 
the interpreter won’t decide a single statement has come to an end until it 
finds the closing bracket (]) that matehes the opening one ([). Typically, the 
end of the line marks the end of a statement in Python, but there 
can be exceptions to this general rule, and multiline lists are just one of them 
(wedl meet the others later). 


Like arrays, lists 
can hold Jata ol 
any type. 


This is 3 y\CW 
vdridblC) ddiicd 
w odds ) wbitb is 
assi^ed 3 list 
odd numbers. 


from datetime import datetime 




[ 1, 

3, 

5, 

7, 

9, 

11, 

13, 

15, 

17, 

19, 

21, 

23, 

25, 

27, 

29, 

31, 

33, 

35, 

37, 

39, 

41, 

43, 

45, 

47, 

49, 

51, 

53, 

55, 

57, 

59 ] 


This is -the list o-f 
odd humbm, ehdlosed 
,h square biradkets. 
This single shalemcni 
extends oveir thvee 
1'mes, whiih is OK. 


There are lots of things that can be done with lists, but we’re going to defer 
any further discussion until a later chapter. All you need to know now is that 
this list now exists, has been assigned to the odds variable (thanks to the use of 
the assignment operator, =), and contains the numbers shown. 


Python variables are dynamically assigned 


Before getting to the next line of code, perhaps a few words are needed about 
variables, especially if you are one of those programmers who might be used 
to predeclaring variables with type information before using them (as is the 
case in statically typed programming languages). 

In Python, variables pop into existence the first time you use them, and their 
type does not need to be predeclared. Python variables take their type 
information from the type of the object they 5 re assigned. In our program, the 
odds variable is assigned a list of numbers, so odds is a list in this case. 

Let 5 s look at another variable assignment statement. As luck would have it, 
this just so happens to also be the next line of code in our program. 


Python comes with all 
the usual operators, 
including <, >, <=, >=, 
==, !=, as well as the 
= assignment operator. 


you are here ► 
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assignment is everywhere 


Ihvokiwg Methods Obtains Results 


The third line of code in our program is another assignment statement. 

Unlike the last one, this one doesn’t assign a data structure to a variable, but instead assigns 
the resuit of a method call to another new variable, called right_this_minute. Take 
another look at the third line of code: 


Heve's axothev 
vaviable V>ei«a 

odds = [ 1, 

3, 

5, 

7, 

9, 

11, 

13, 

15, 

17, 

19, 

dv-eated 3»d 

21, 

23, 

25, 

27, 

29, 

31, 

33, 

35, 

37, 

39, 

assigned 3 value- 

41, 

43, 

45, 

47, 

49, 

51, 

53, 

55, 

57, 

59 ] 



from datetime import datetime 


right this minute = datetime.today().minute 


Y. 


if right_this_minute in odds: 

print( M This minute seems a little odd. M ) 
else: 

print( M Not an odd minute.") 


This tali gthcra-tcs 
a value *to assign -to 

the variable. 


Iwvokiwg built-m module fuwctiowality 

The third line of code invokes a method called today that comes with the datetime 
submodule, which is itself part of the datetime module (we did say this naming strategy was 
a little confusing). You can teli today is being invoked due to the Standard postfix parentheses: 
(). 

When today is invoked, it returns a “time object,” which contains many pieces of information 
about the current time. These are the current time 5 s attributes, which you can access via the 
customary dot-notation syntax. In this program, we are interested in the minute attribute, 
which we can access by appending . minute to the method invocation, as shown above. The 
resulting value is then assigned to the right_this_minute variable. You can think of this 
line of code as saying: create an object that represents today 3 s time, then extract the value of the minute 
attribute before assigning it to a variable. It is tempting to split this single line of code into two lines 
to make it “easier to understand,” as follows: 


You’ll see 
more ol the 
dot-notation 
syntax later 
in this hook. 


f-ivst, detevVme -the 
tuv-vewt • 


time_now = datetime.today() 

right this minute = time now.minute 


•then extv-adt the 
*»inute value- 


You can do this (if you like), but most Python programmers prefer not to create the temporary 
variable (time_now in this example) unless it’s needed at some point later in the program. 
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Peciding When to Ruw Plocks of Code 


At this stage we have a list of numbers called odds. We also have a minute value 
called right_this_minute. In order to work out whether the current minute 
value stored in right_this_minute is an odd number, we need some way of 
determining if it is in the odds list. But how do we do this? 

It turns out that Python makes this type of thing very straightforward. As well 
as including all the usual comparison operators that you’d expect to find in any 
programming language (such as >, <, >=, <=, and so on), Python comes with a few 
“super” operators of its own, one of which is in. 


The in operator checks if one thing is inside another. Take a look at the next line of 
code in our program, which uses the in operator to check whether right_this_ 
minute is inside the odds list: 


This "it" statewwt 
v,ill evaWaie U 

«W «r 'W- 


right this minute = datetime.today().minute 


if right this minute in odds: 


print( M This minute seems a little odd. M ) 


The w ih w operator 
is power-Pul. It dan 
determine whether 
ohe thing is inside 
dnother. 


The in operator returns either True or False. As you’d expect, if the value in 
right_this_minute is in odds, the if statement evaluates to True, and the 
block of code associated with the i f statement exeeutes. 


Blocks in Python are easy to spot, as they are always indented. 


In our program there are two blocks, which each contain a single call to the print 
function. This function can display messages on screen (and well see lots of uses of it 
throughout this book). When you enter this program code into the edit window, you 
may have noticed that IDLE helps keep you straight by indenting automatically. This 
is very useful, but do be sure to check that IDLEs indentation is what you want: 


Here is one 
blodk ot dode- 
Kote : “the dode 
is indented- 


right this minute = datetime.today().minute 


if right_this_minute in odds: 

print("This minute seems a little odd.") 


else: 


The "prini" tundiion 
displays a message on 

standav-d outpui Ci-c-, 
yowr sdreen). 


print("Not an odd minute.") 



Did you notice that there are no curly braces here? 


And here is anoiher loUk ot tode- 
Noie: »-t's indeniedi W 


you are here t 
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no curly braces 


What Happened to My Curly Praces? 


If you are used to a programming language that uses curly braces ({ and }) 
to delimit blocks of code, encountering blocks in Python for the first time can 
be disorienting, as Python doesn’t use curly braces for this purpose. Python 
uses indentation to demarcate a block of code, which Python programmers 
prefer to call suite as opposed to block (just to mix things up a little). 

It’s not that curly braces don’t have a use in Python. They do, but—as we’ll 
see in Ghapter 3—curly braces have more to do with delimiting data than 
they have to do with delimiting suites (i.e., blocks) of code. 

Suites within any Python program are easy to spot, as they are always 
indented. This helps your brain quickly identify suites when reading code. 
The other visual clue for you to look out for is the colon character (:), which 
is used to introduce a suite that’s associated with any of Python 5 s control 
statements (such as if, else, for, and the like). Youil see lots of examples 
of this usage as you progress through this book. 

A colon introduces an indented suite of code 

The colon (:) is important, in that it introduces a new suite of code that must 
be indented to the right. If you forget to indent your code after a colon, the 
interpreter raises an error. 

Not only does the if statement in our example have a colon, the else has 
one, too. Here’s all the code again: 


Instead of referring 
to a cocte "hlock,” 
Python programmers 
use the word "suite.” 
Both names are used 
in practice, hut the 
Pytho n docs prefer 

W • l 11 

suite. 


from datetime import datetime 

19, 
39, 
59 ] 

right_this_minute = datetime.today().minute 

if right_this_minute in odds: 

print("This minute seems a little odd. M ) 
else: 

print( M Not an odd minute.") 



[ 1, 

3, 

5, 

7, 

9, 

11, 

13, 

15, 

17 

21, 

23, 

25, 

27, 

29, 

31, 

33, 

35, 

37 

41, 

43, 

45, 

47, 

49, 

51, 

53, 

55, 

57 


Colons ‘mtrodude 
indented suites. 


We’re nearly done. There 5 s just one final statement to discuss. 


16 Chapter 1 










the basies 


What "else" Can Yoo Have with "if"? 


We are nearly done with the code for our example program, in that there is 
only one line of code left to discuss. It is not a very big line of code, but it 5 s 
an important one: the e Ise statement that identifles the block of code that 
exeeutes when the matehing if statement returns a False value. 

Take a closer look at the e Ise statement from our program code, which we 
need to unindent to align with the i f part of this statement: 


£ee “the dolon? ^ 


pid you 

that the "else" is 

unmdented to ali^ 
W,th the "it"? 



if right_this_minute in odds: 

print( M This minute seems a little odd. M ) 
else: 

print( M Not an odd minute.") 



It is a very common 
slip-up lor Python 
newhies to lorgfet 
the colon when 
lirst writingf code. 


Neither. Python spells it elif. 

If you have a number of conditions that you need to 
check as part of an i f statement, Python provides 
elif as well as else. You can have as many elif 
statements feach with its own suite) as needed. 


Here 5 s a small example that assumes a variable 
called today is previously assigned a string 
representing whatever today is: 

if today == 'Saturday': 

print('Party! ! ' ) 
elif today == 'Sunday': 

print(’Recover.') 
else: 

print('Work, work, work. 1 ) 


Thlree ihdividual 
sui-fces: Ohe -for 

tov the “elit”, 
3hd the -final 
dat<ih—all -fov 
the "else". 
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indent like crazy 


Suites Can Contam Embedded Suites 

Any suite can contain any number of embedded suites, which also have to be 
indented. When Python programmers talk about embedded suites, they tend 
to talk about levels of indentation. 

The initial level of indentation for any program is generally referred to as the 
Jirst or (as is so common when it comes to counting with many programming 
languages) indentation level zero. Subsequent levels are referred to as the 
second, third, fourth, and so on (or level one, level two, level three, and so on). 

Here 5 s a variation on the today example code from the last page. Note how 
an embedded if /eIse has been added to the if statement that executes 
when today is set to 'Sundayf We’re also assuming another variable called 
condition exists and is set to a value that expresses how you’re currently 
feeling. WeVe indicated where each of the suites is, as well as at which level of 
indentation it appears: 



level one 


These sin 
lines o-p 
are boih 
suites. 


dode 


It is important to note that code at the same level of indentation is only 
related to other code at the same level of indentation if all the code appears 
within the same suite. Otherwise, they are in separate suites, and it does 
not matter that they share a level of indentation. The key point is that 
indentation is used to demarcate suites of code in Python. 
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What We Already Kwow 

With the final few lines of code discussed, let 5 s pause to review what 
the odd. py program has told us about Python: 



BULLET POINTS 


■ Python comes with a built-in IDE called IDLE, which 
lets you create, edit, and run your Python code—all 
you need to do is type in your code, save it, and then 
press F5. 

■ IDLE interacts with the Python interpreter, which 
automates the compile-link-run process for you. This 
lets you concentrate on writing your code. 

■ The interpreter runs your code (stored in a file) from 
top to bottom, one line at a time. There is no notion of 
a main () function/method in Python. 

■ Python comes with a powerful Standard library, which 
provides access to lots of reusable modules (of which 
datet ime is just one example). 

■ There is a collection of Standard data structures 
available to you when you're writing Python 
programs. The list is one of them, and is very similar 
in notion to an array. 


The type of a variable does not need to be declared. 
When you assign a value to a variable in Python, it 
dynamically takes on the type of the data it refers to. 

You make decisions with the if/elif/else 
statement. The if, elif , and else keywords 
precede blocks of code, which are known in the 
Python world as “suites.” 

It is easy to spot suites of code, as they are always 
indented. Indentation is the only code grouping 
mechanism provided by Python. 

In addition to indentation, suites of code are also 
preceded by a colon (:). This is a syntactical 
requirement of the language. 




Let’s extend this program to do more. 

It’s true that we needed more lines to describe what this short 
program does than we actually needed to write the code. But 
this is one of the great strengths of Python :you can get a lot 
done with a few lines of code. 

Review the list above once more, and then turn the page to 
make a start on seeing what our program 5 s extensions will be. 

you are here ► 
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now what? 


Extewdiwg Our Program to Po More 

Let 5 s extend our program in order to learn a bit more Python. 

At the moment, the program runs once, then terminates. Imagine that we 
want this program to exeeute more than once; let 5 s say five times. Specifically, 
let 5 s exeeute the “minute checking code” and the if/else statement five 
times, pausing for a random number of seconds between each message 
display (just to keep things interesting). When the program terminates, five 
messages should be on screen, as opposed to one. 


Here’s the code again, with the code we want to run multiple times circled: 



What we need to do: 


o 

o 

o 


Loop over the encircled code. 

A loop lets us iterate over any suite, and Python provides a number of ways to do just that. In this 
case (and without getting into why), we’11 use Python’s for loop to iterate. 

Pause exeeution. 

Python’s Standard time module provides a function called sleep that can pause exeeution for an 
indicated number of seconds. 

Generate a random number. 

Happily, another Python module, random, provides a function called randint that we can use 
to generate a random number. Let’s use randint to generate a number between 1 and 60, then 
use that number to pause the exeeution of our program on each iteration. 


We now know what we want to do. But is there a 
preferred way of going about making these changes? 
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What'$ the Pest Approach to Solving 
This Problew? 


You know what you need to do: put 
your head down, read the docs, and 
work out the Python code you need 
to solve this problem. When youve 
done this, youre ready to change 
your program as needed... 



That approach works, but I'm more 
of an experimenter myself. I like to 
try out small snippets of code before 
I commit to making changes to my 
working program. I'm happy to read 
the docs, but like to experiment too... 


3 u\r cl 


Poth approaches work with Python 


You can follow both of these approaches when working with Python, but most 
Python programmers favor experimentation when trying to work out what 
code they need for a particular situation. 

Don’t get us wrong: we are not suggesting that Bob 5 s approach is wrong and 
Laura 5 s is right. It’s just that Python programmers have both options available 
to them, and the Python Shell (which we met briefly at the start of this 
chapter) makes experimentation a natural choice for Python programmers. 

Let 5 s determine the code we need in order to extend our program, by 
experimenting at the >>> prompt. 


Experimenting at 
the »> prompt 
kelps you work out 
the code you need. 
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Shell experiments 


Returwmg to the Python Shell 

Here’s how the Python Shell looked the last time we interacted with it (yours might 
look a little different, as your messages may have appeared in an alternate order): 


# # # Python 3.4.3 Shell 

Python 3.4.3 (v3.4.3:9b73fIc3e601, Feb 23 2015, 02:52:03) 

[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin 

Type "Copyright", "credits" or "license()" for more information. 

»> -RESTART- 

»> 

This minute seems a little odd. 

»> -RESTART- 

»> 

Not an odd minute. 

»> 



Ln: 10 Coi: 4 


The Python Shell (or just “shell” for short) has displayed our progranTs messages, but 
it can do so much more than this. The >>> prompt allows you to enter any Python 
code statement and have it exeeute immediately . If the statement produces output, the 
shell displays it. If the statement results in a value, the shell displays the calculated 
value. If, however, you create a new variable and assign it a value, you need to enter 
the variable’s name at the >>> prompt to see what value it contains. 

Gheck out the example interactions, shown below. It is even better if you follow along 
and try out these examples at your shell. Just be sure to press the Enter key to terminate 
each program statement, which also telis the shell to exeeute it now : 


# # # Python 3.4.3 Shell 

Python 3.4.3 (v3.4.3:9b73fIc3e601, Feb 23 2015, 02:52:03) 

[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin 

Type "Copyright", "credits" or "license()" for more information. 

»> -RESTART- 

»> 

This minute seems a little odd. 

»> -RESTART- 

»> 

Not an odd minute. T 

»> ^_shell displays a .«essage on sdv-een as a resuit ot -this 

Hello^um' ' H6ll ° MUm! ) t<>de exeduting (do* i tov-get to press £*{&). 

._ 2 i+?i r It you per-Por» a daldulation, the shell displays the 

42 ^ vesultm^ value (a-ftev- you pvess Errtev-). 

»> ultimate_answer = 21+21 ^ssigning a value j» a vav-iable does not display the 

»> ultimate_answer J ^-vaviable s value- Y ou bave to syedi-ritally ask tne 

sKcll “to do so- 

»> 


Ln: 20 Coi: 4 
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Experimenting at the Shell 

Now that you know you can type a single Python statement into the >>> prompt 
and have it exeeute immediately, you can start to work out the code you need to 
extend your program. 

Here’s what you need your new code to do: 

Loop a specified number of times. WeVe already decided to use 
Python’s for loop here. 

Pause the program for a specified number of seconds. The sleep 
function from the Standard library 5 s time module can do this. 

Generate a random number between two provided values. The 
randint function from the random module will do the trick. 


□ 

□ 

□ 


Rather than continuing to show you complete IDLE screenshots, we’re only 
going to show you the >>> prompt and any displayed output. Specifically, from 
this point onward, you’11 see something like the following instead of the earlier 
screenshots: 


The shell promat 





»> print( f Helio Mum! f ) 
Helio Mum! 


e 



The output resultmj 'f' rorr ' e%e£utmg 
-the smjle £odc state^en!) whieh is 
shovm m blue \y\ your shell 


Over the next few pages, we’re going to experiment to figure out how to add 
the three features listed above. Weil play with code at the >>> prompt until we 
determine exactly the statements we need to add to our program. Leave the odd. 
py code as is for now, then make sure the shell window is active by selecting it. 
The cursor should be blinking away to the right of the »>, waiting for you to 
type some code. 

Flip the page when you’re ready. Let the experiments begin. 


you are here ► 
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Iteratiwg Over a Sequence of Objects 


We said earlier that we were going to employ Python’s for loop here. The 
for loop is perfect for controlling looping when you know ahead of time how 
many iterations you need. (When you don’t know, we recommend the while 
loop, but we’ll save discussing the details of this alternate looping construet 
until we actually need it). At this stage, all we need is for, so let’s see it in 
action at the >>> prompt. 

We present three typical uses of for. Let’s see which one best fits our needs. 


Use "for” when 


loo 


k 


pingf a known 
number ol times. 


Usage example 1. This for loop, below, takes a list of numbers and 
iterates once for each number in the list, displaying the current number on 
screen. As it does so, the for loop assigns each number in turn to a loop 
iteration variable , which is given the name i in this code. 


As this code is more than a single line, the shell indents automatically for you 
when you press Enter after the colon. To signal to the shell that you are done 
entering code, press Enter twice at the end of the loop’s suite: 


»> for i in [ 1 , 2, 

print(i) 


1 

2 

3 


3 ] 



m 


We .EEd V as Uf .tssaW 
tte «»*.*, Wt« uM «; 

about ar>ytKm<y ttavm$ said that, 1 » J ’ 
awd V are mfcredibly f^ular ar«o»5 ">ost 
pvoyan>n>ers m this situation- 


As this is a suite, you heed to press +u, e i , 


Note the indentation and colon. Like if statements, the code associated with a 
for statement needs to be indented. 


Usage example 2. This for loop, below, iterates over a string, with 
each character in the string being processed during each iteration. This 
works because a string in Python is a sequence. A sequence is an ordered 
collection of objects (and we’ll see lots of examples of sequences in this book), 
and every sequence in Python can be iterated over by the interpreter. 

»> for ch in "Hi!" 

print(ch) 



A secruence is an 
ordered collection 
ol objects. 


H 

■ 

1 

V 


Pytho A smart e»ou 5 h to v-ork ^thatth 
strms should be iterated over o^e-tharatter 

ataWa«dthats*,hy*eused 6h as 

-the Vdfiablc 


Nowhere did you have to teli the for loop how big the string is. Python is smart 
enough to work out when the string ends , and arranges to terminate (i.e., end) 
the for loop on your behalf when it exhausts all the objects in the sequence. 
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Iterating a Specific Number of Times 

In addition to using f or to iterate over a sequence, you can be more exact 
and specify a number of iterations, thanks to the built-in function called 
range. 

Let’s look at another usage example that showcases using range. 

Usage example 3. In its most basic form, range accepts a single integer 
argument that dictates how many times the for loop runs (we’ll see other 
uses of range later in this book). In this loop, we use range to generate a 
list of numbers that are assigned one at a time to the num variable: 

»> for num in range (5) : 

print( 1 Head First Rocks! 1 ) 

Head First Rocks! 

Head First Rocks! 

Head First Rocks! 

Head First Rocks! 

Head First Rocks! 

The for loop didnd use the num loop iteration variable anywhere in the 
loop’s suite. This did not raise an error, which is OK, as it is up to you (the 
programmer) to decide whether or not num needs to be processed further in 
the suite. In this case, doing nothing with num is fine. 


Wt 3sk ' a f 3 r 3 "9« of fi ve nuftbm, 

S ° c we ***** five W, whidh resuit; 

T +,ve , "*“ 3 9«- press E.-fcer 

tw.ee to ruh dode that has a suite 



Indeed we are. Task #1 is complete. 

The three usage examples show that Python’s for loop is 
what we need to use here, so lef s take the technique shown 
in Usage example 3 and use it to iterate a specified number 
of times using a for loop. 
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make that change 


Applyiwg the Outcowe of Task #1 to Our Code 

Here’s how our code looked in IDLE 5 s edit window before we worked on Task #1: 


# ($ # odd.py -/Users/paul/Desktop/NewBook/ch01/odd.py (3.5.1) 


from datetime import datetime 


odds = [1, 

3, 

5, 

7, 

9, 

11, 

13, 

15, 

17, 

19, 

H 

CM 

23, 

25, 

CM 

29, 

31, 

33, 

35, 

37, 

39, 

41, 

43, 

45, 

47, 

49, 

51, 

53, 

55, 

57, 

59] 


right this minute = datetime.today().minute 



if right_this_minute in odds: \ 

print ("This minute seems a little odd.") 
else: } 

print ( "Not an odd minute." ) 


Ln: 1 4 Coi: 0 


This is thc 
£ode wc 
wdh-fc "to 

rep eat 


You now know that you can use a for loop to repeat the five lines of code at the 
bottom of this program five times. The five lines will need to be indented under the 
for loop, as they are going to form the loop’s suite. Specifically, each line of code 
needs to be indented once. However, don’t be tempted to perform this action on each 
individual line. Instead, let IDLE indent the entire suite for you in one go. 

Begin by using your mouse to select the lines of code you want to indent: 


# ® # odd.py -/Users/paul/Desktop/NewBook/ch01/odd.py (3.5.1) 


from datetime import datetime 


odds = [1, 

3, 

5, 

7, 

9, 

11, 

13, 

15, 

17, 

19, 

H 

CM 

23, 

25, 

27, 

29, 

31, 

33, 

35, 

37, 

39, 

41, 

43, 

45, 

47, 

49, 

51, 

53, 

55, 

57, 

59] 


right_this_minute = datetime.today().minute 

if right_this_minute in odds: 

print("This minute seems a little odd.") 
else: 

print("Not an odd minute.") 


Ln: 14 Coi: 0 


l td “bhe 

E M # 
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Indent Suites with Format...lndent Region 

With the five lines of code selected, choose Indent Region from the Format menu in 
IDLEs edit window. The entire suite moves to the right by one indentation level: 


# # # *odd.py - /Users/paul/Desktop/NewBook/ch01/odd.py (3.5.1)* 


from datetime import datetime 


odds = [1, 

3, 

5, 

7, 

9, 

11, 

13, 

15, 

17, 

19, 

H 

CM 

23, 

25, 

27, 

29, 

31, 

33, 

35, 

37, 

39, 

41, 

43, 

45, 

47, 

49, 

51, 

53, 

55, 

57, 

59] 


right_this_minute = datetime. today () .minute 

if right_this_minute in odds: 

print( M This minute seems a little odd.") 


else: 

print( M Not an odd minute.") 





Ln: 14 Coi: 0 


The |r\derrt Region 
option -from *the 
Format menu 
indenis ali ot the 
seledted lines ot 
£ode in eme 50. 



Note that IDLE also has a Dedent Region menu option, which unindents suites, and 
that both the Indent and Dedent menu commands have keyboard shortcuts, which differ 
slightly based on the operating system you are running. Take the time to learn the 
keyboard shortcuts that your system uses now (as you’11 use them all the time). With the 
suite indented, it’s time to add the for loop: 


# (§ # odd.py - /Users/paul/Desktop/NewBook/ch01 /odd.py (3.5.1) 


from datetime import datetime 


Add -the 

ue )) 

-ror 

loop lihc. 


odds = [1, 

3, 

5, 

7, 

9, 

11, 

13, 

15, 

17, 

19, 

21, 

23, 

25, 

27, 

29, 

31, 

33, 

35, 

37, 

39, 

41, 

43, 

45, 

47, 

49, 

51, 

53, 

55, 

57, 

59] 



The w W* 

loop s suite 
is properiy 
'mdented- 



for i in range (5): 

right_this_minute = datetime. today () .minute 

if right_this_minute in odds: 

print("This minute seems a little odd." 
else : 

print("Not an odd minute.") 


Ln: 15 


) 


Coi: 0 
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feeling sleepy? 


Arrawgmg to Pause Execution 

Let’s remind ourselves of what we need this code to do: 

Loop a specified number of times. 

Pause the program for a specified number of seconds. 

Generate a random number between two provided values. 





We’re now ready to return to the shell and try out some more code to help 
with the second task: pause the program for a specified number of seconds. 

However, before we do that, recall the opening line of our program, which 
imported a specifically named function from a specifically named module: 


from datetime import datetime 



This is one way to import a function into your program. Another equally 
common technique is to import a module without being specific about the 
function you want to use. Let’s use this second technique here, as it will 
appear in many Python programs you’11 come across. 


I his usage o\ u ir»po* 
birihgs ih -the hamed 
tun^tioh to your py*c 
theh ihVoke 

without usihg -the dc 
notation syh-tax. 


As mentioned earlier in this chapter, the sleep function can pause exeeution 
for a specified number of seconds, and is provided by the Standard library’s 
time module. Let’s import the module first , without mentioning sleep just 
yet: 

»> import time 

>» 


TWis telis tV>e sKell to 
i^ovt tKe "tWe module- 


When the import statement is used as it is with the time module above, 
you get access to the facilities provided by the module without anything 
expressly named being imported into your progranTs code. To access a 
function provided by a module imported in this way, use the dot-notation 
syntax to name it, as shown here: 


This is -the humber o-f 


scdohds to sleep -for 


|sfdv^e the module 
Jftv-st (be-fore 
the period)- 


»> time. s leep (5) 

»> 


Note that when you invoke sleep in this way, the shell pauses for five 
seconds before the >>> prompt reappears. Go ahead, and try it now. 


Speei£y the -Curetio* you wa*t 
to mvoke (a-fter the period). 
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Importation Confusion 



Hang on a second ...Python supports two 
importation mechanisms? Doesn't that get 
kind of confusing? 


That’s a great question. 

Just to be ciear, there aren’t two importation mechanisms in 
Python, as there is only one import statement. However, the 
import statement can be used in two ways. 

The first, which we initially saw in our example program, 
imports a named function into our progranTs namespace, 
which then allows us to invoke the function as necessary without 
having to link the function back to the imported module. (The 
notion of a namespace is important in Python, as it defines the 
context within which your code runs. That said, we’re going to 
wait until a later chapter to explore namespaces in detail). 

In our example program, we use the first importation technique, 
then invoke the datetime function as datetime () ,not as 
datetime.datetime(). 

The second way to use import is to just import the module, as 
we did when experimenting with the time module. When we 
import this way, we have to use the dot-notation syntax to access 
the module’s functionality, as we did with time . sleep () . 


tPrejure n? 

Dumb Questions 


Is there a correct way to use import? 

It can often come down to personal preference, as some programmers like to be very specific, while others doni However, there is a 
situation that occurs when two modules (we’ll call them A and B) have a function of the same name, which we ll call F. If you put f r om A 
import Fandfrom B import F in your code, how is Python to know which F to invoke when you call F ()? The only way you 
can be sure is to use the nonspecific import statement (that is, put import A and import B in your code), then invoke the specific 
F you want using either A. F () or B. F () as needed. Doing so negates any confusion. 
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every now and again 


(reweratmg Random Integers with Python 


Although it is tempting to add import time to the top of our program, then 
call time . sleep (5) in the for loop’s suite, we aren’t going to do this right 
now. We aren’t done with our experimentations. Pausing for five seconds isn’t 
enough; we need to be able to pause for a random amount of time. With that in 
mind, let’s remind ourselves of what weVe done, and what remains: 


& 


Loop a specified number of times. 


Pause the program for a specified number of seconds. 


Generate a random number between two provided values. 


Once we have this last task completed, we can get back to confidently changing 
our program to incorporate all that weVe learned from our experimentations. 
But we’re not there yet—let’s look at the last task, which is to generate a random 
number. 

As with sleeping, the Standard library can help here, as it includes a module called 
random. With just this piece of information to guide us, let’s experiment at the 
shell: 


»> import random 

»> 

Now what? We could look at the Python docs or consuit a Python reference 
book...but that involves taking our attention away from the shell, even though it 
might only take a few moments. As it happens, the shell provides some additional 
functions that can help here. These functions aren’t meant to be used within 
your program code; they are designed for use at the >>> prompt. The first is 
called dir, and it displays all the attributes associated with anything in Python, 
including modules: 


'NV MAGICCONST 


»> dir (random) 

[ 1 BPF 1 , 1 LOG4 1 , 

'Random 1 , ... ^randint' 

1 sample 1 , 1 seed 1 , 1 setstate 

f uniform f , 1 vonmisesvariate 


Use "dir” to 

cruery an 
okject. 


Bwried in -the middle of this 
H list is the hame of the 
■riihdtioh we need- 


'RECIP_BPF', 
random', 'randrange', 

'shuffle', 'triangular' 
'weibullvariate'] 


This list has a lot in it. Of interest is the randint () function. To learn more 
about randint, let’s ask the shell for some help. 


Ttis is ah abiridaed 
list- What you'l| see 

y Sdheeh is mudh 
lohgcv. 
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Askiwg the Interpreter for Help 

Once you know the name of something, you can ask the shell for help. When 
you do, the shell displays the section from the Python docs related to the name 
you’re interested in. 

Let 5 s see this mechanism in action at the >>> prompt by asking for help with 
the randint function from the random module: 


»> help (random. randint) 

Help on method randint in module random: 



Use "Mp” 
to read the 
Pytho n docs. 


/\sk -for helf at 
-the »> pv-om^t- 




randint(a, b) method of random.Random instance 

Return random integer in range [a, b], including 
both end points. 


• ahd see the assodiated 
do£ur*eirfcatioK right ih the shell- 


A quick read of the displayed docs for the randint function confirms what 
we need to know: if we provide two integers to randint, we get back a 
random integer from the resulting inclusive range. 

A few final experiments at the >>> prompt show the randint function in 
action: 

»> random. randint (1,60) 

27 

»> random. randint (1,60) 

34 

»> random. randint (1,60) 

46 



You can recall the last 
command(s) typed into 
the IDLE >>> prompt 
by typing Alt-P when 
using Linux or Windows. 
On Mac OSX, use Ctrl-P. 
Think of the "P"as 
meaning "previous." 



I-P you Ve tollowing along, what you'll 
see oh you\r sdveeh will vary, as the 
integeirs vetuvned by Vandint" ave 
genevated vandomly. 


Betause vou imPovted the Vavdom" module us,^ Vyovt vavdom , you 
!eed to vemembev to ? vet.* the tali to VaAt with the module «me 
and a dot. So it's Vandom-vandinto and not vandmtC^ ■ 

With this, you are now in a position to place a satisfying check mark against 
the last of our tasks, as you now know enough to generate a random number 
between two provided values: 

0 Generate a random number between two provided values. 


It’s time to return to our program and make our changes. 
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what we now know 


Reviewing Our Experiments 

Before you forge ahead and change your program, let’s quickly review the 
outcome of our shell experiments. 

We started by writing a for loop, which iterated five times: 

»> for num in range(5) : 

print('Head First Rocks!') 


Head First Rocks! 
Head First Rocks! 
Head First Rocks! 
Head First Rocks! 
Head First Rocks! 



W aske . d a oP -five numbevs, 
so -five times, which 

^sults ih -five messages. 


Then we used the sleep function from the time module to pause exeeution 
of our code for a specified number of seconds: 


»> import time _ s v, e || i^ovb -the “W" «module, 

»> time. sleep (5) ^ m voke the "sleef” Pwfrbon- 


And then we experimented with the randint function (from the random 
module) to generate a random integer from a provided range: 


»> import random 

»> random. randint (1,60) 

12 

»> random. randint (1,60) 
42 

»> random. randint (1,60) 
17 


di-Pteveiot imtegers ave 9 e*evated 

^ ot)te m< * re > as Vandint" vetuvus a di-PPeven 
vandom mte 9 ev each time it's mvoked- 


We can now put all of this together and change our program. 

Let 5 s remind ourselves of what we decided to do earlier in this chapter: have 
our program iterate, exeeuting the “minute checking code” and the i f / 
e Ise statement five times, and pausing for a random number of seconds 
between each iteration. This should resuit in five messages appearing on 
screen before the program terminates. 
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Code Experiments Magnets 

Based on the specificatiori at the bottom of the last page, as well as 
the results of our experimentations, we went ahead and did some 
of the required work for you. But, as we were arranging our code 
magnets on the fridge (don't ask) someone slammed the door, and 
now some of our code's all over the floor. 


Your job is to put everything back together, so that we can run the 
new version of our program and confirm that it's working as required. 


fetide wViifi-Vi fcode 
r*ay\e-t ^oes m tat\\ 
o( -the dashed-lme 
lota-tions. 


right_this_minute = datetime.today().minute 
if right_this_minute in odds: 

print( M This minute seems a little odd. M ) 
else: 

print( M Not an odd minute.") 
wait time = 

( ) 


from datetime import datetime 



odds = [ 1, 

3 , 

5, 

7, 

9, 

11, 

13, 

15, 

17, 

19, 

21, 

23, 

25, 

27, 

29, 

31, 

33, 

35, 

37, 

39, 

41, 

43, 

45, 

47, 

49, 

51, 

53, 

55, 

57, 

59 ] 


Wheve do 
all -these. 
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rearranged code 



Code Experiments Magnets Solution 

Based on the specificatiori from earlier, as well as the results of our 
experimentations, we went ahead and did some of the required work 
for you. But, as we were arranging our code magnets on the fridge 
(don't ask) someone slammed the door, and now some of our code's all 
over the floor. 


v/ow dont Kave to 
Pwt youV 'im^ovb 

at the to? 7 0tt ' r 
t ode, but it «s a 
NwelUestablished 


tonventio' r > a»«on$ 

Python Y ro V' 

&o so- 



Your job was to put everything back together, so that we could run the 
new version of our program and confirm that it's working as required. 


from datetime import datetime 


import random 


import time 

V. 


The "W 

loo? itev-ates 

£)<ACTL,Y 

-f We *b»w»cs- 



The 'Vandint" WW 
pv-ovides a va^dorr. 

integer that i» v&r** 
to a new vaviable talled 
'WitjW» xhifch— 


odds = [ 1, 

3, 

5, 

7, 

9, 

11, 

13, 

15, 

17, 

19, 

21, 

23, 

25, 

27, 

29, 

31, 

33, 

35, 

37, 

39, 

41, 

43, 

45, 

47, 

49, 

51, 

53, 

55, 

57, 

59 ] 


for i in range(5): 


] 



right_this_minute = datetime.today().minute 
if right_this_minute in odds: 

print("This minute seems a little odd.") 
else: 

print("N ot an odd minute.") 
wait time = 


time. sleep_ T 


random.randint(1, 60) 
; I wait time 


) 



7 ' S theh use< ? '* *** «II to "sleep" to pause 


AII o-f -this £ode 
' s indented 
■ under the “tov" 
statemeht, as it 
'* all part o-f the 
"tv" statements 
suite- ^emember-' 
Python does «ot 
use durly brades to 
delimit suites; it 
uses indentation 
instead- 


34 


Chapter 1 



























the basies 



Let’s try running our upgraded program in IDLE to see what happens. Change your version 
of odd. py as needed, then save a copy of your new program as odd2. py. When you’re 
ready, press F5 to exeeute your code. 


IA/hen ?' ress ^ ■** 
v-uv\ ttos toA e- 



-y° u see ou-tpu-t 

similae io ibis. Jusi 

rer»er»be\r -tha-t your 

output will di-WW, as 
"tbc r 3hdorv> hUmbcV"S youV" 
pV-og\ram gemeira-fces 
Iikely woh't r»a-kh ours. 



# # # odd2.py -/Users/Paul/Desktop/_NewBook/ch01/odd2.py (3.4.3) 


from datetime import datetime 


import random 
import time 


odds = [ 1, 

3, 

5, 

7, 

9, 

11, 

13 / 

15, 

17, 

19, 

21, 

23, 

25, 

27, 

29, 

31, 

33, 

35, 

37, 

39, 

41, 

43, 

45, 

47, 

49, 

51, 

53, 

55, 

57, 

59 ] 


for i in range(5): 

right_this_minute = datetime.today().minute 
if right_this_minute in odds: 

print( This minute seems a little odd. ) 
else: 

print( Not an odd minute.) 

wait_time = random.randint(1, 60) 
time.sleep(wait_time) 


Ln: 19 Coi: 0 


# # # Python 3.4.3 Shell 

»> -RESTART- 

»> 

This minute seems a little odd. 

This minute seems a little odd. 

Not an odd minute. 

Not an odd minute. 

Not an odd minute. 

»> 



Ln: 25 Coi: 4 



t viovvy rf you see a diWerewt Itet «eoays 
those Ibo*,* here- You should see *We «essa^es, 
,ai's bow w>a»7 -times the loof> Code vuns. 
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update our list 


Updating What We Already Know 

With odd2 . py working, let’s pause once more to review the new 
things we’re learned about Python from these last 15 pages: 



BULLET POINTS 


■ When trying to determine the code that they 
need to solve a particular problem, Python 
programmers often favor experimenting with code 
snippets at the shell. 


If you need to pause the execution of your 
program for a specified number of seconds, use 
the sleep function provided by the Standard 
library’s time module. 


If you’re looking at the »> prompt, you’re at the 
Python Shell. Go ahead: type in a single Python 
statement and see what happens when it runs. 

The shell takes your line of code and sends it to 
the interpreter, which then executes it. Any results 
are returned to the shell and are then displayed 
on screen. 

The for loop can be used to iterate a fixed 
number of times. If you know ahead of time how 
many times you need to loop, use for. 

When you don’t know ahead of time how often 
you’re going to iterate, use Python’s while loop 
(which we have yet to see, but—doni worry—we 
will see it in action later). 

The for loop can iterate over any sequence 
(like a list or a string), as well as execute a fixed 
number of times (thanks to the range function). 


■ You can import a specific function from a module. 
Forexample, from time import sleep 
imports the sleep function, letting you invoke it 
without qualification. 

■ If you simply import a module—for example, 
import time—you then need to qualify the 
usage of any of the module’s functions with the 
module name, like so: time. sleep (). 

■ The random module has a very useful function 
called randint that generates a random 
integer within a specified range. 

■ The shell provides two interactive functions that 
work at the »> prompt. The dir function lists 
an objecfs attributes, whereas help provides 
access to the Python docs. 


iliereiare no 

Dumb Oiiestions 


O: 


Do I have to remember all this stuff? 


No, and don’t freak out if your brain is resisting the insertion of everything seen so far. This is only the first chapter, and we’ve designed it 
to be a quick introduction to the world of Python programming. If you’re getting the gist of what’s going on with this code, then you’re doing fine. 
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A Few Lines of Code Po a Lot 


0 O 




4 





/ 


It is, but we are on a roll here. 

It’s true weVe only touched on a small amount 
of the Python language so far. But what weVe 
looked at has been very useful. 

What weVe seen so far helps to demonstrate 
one of Python’s big selling points: a few lines of 
code do a lot. Another of the language 5 s claims 
to fame is this: Python code is easy to read. 

In an attempt to prove just how easy, we 
present on the next page a completely different 
program that you already know enough about 
Python to understand. 

Who 5 s in the mood for a nice, cold beer? 
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serious about beer 


Codiwg a Serious Pusiwess Application 

With a tip of the hat to Head First Java, let 5 s take a look at the Python version of 
that classic’s first serious application: the beer song. 

Shown below is a screenshot of the Python version of the beer song code. Other 
than a slight variation on the usage of the range function (which we’ll discuss in 
a bit), most of this code should make sense. The IDLE edit window contains the 
code, while the tail end of the progranTs output appears in a shell window: 





beer song. py - /Users/'Paul/Desktop/NewBook/chOi/beersong.py (3,4,3) 


word - "bottles" 

for beer_num in range (99, 0, -1): 

print (beer_num, word, "of beer on the wall. ) 

print (beer_num, word, "of beer.) 

print ( Take one down. ' ) 

print ( Pass it around. IT ) 

if beer_num == I: 

print ("No more bottles of beer on the wall. 11 ) 
else : 

new_num = beer_num - 1 

if new_num == 1: 

word = "bottle 11 

print (new_num, word, "of beer on the wall. 1 ) 
print () 

# O Python 3.4.3 Shell 



ftunning -this tode 
pvodudes -this 
ou-tpu-t in -the 
shell. 



Pealiwg with all that beer... 

With the code shown above typed 
into an IDLE edit window and saved, 
pressing F5 produces a lot of output in 
the shell. WeVe only shown a little bit 
of the resulting output in the window 
on the right, as the beer song starts 
with 99 bottles of beer on the wall and 
counts down until there’s no more beer. 
In fact, the only real twist in this code 
is how it handles this “counting down,” 
so let’s take a look at how that works 
before looking at the progranTs code in 
de tail. 


3 bottles of beer on the wall. 

3 bottles of beer. 

Take one down. 

Pass it around. 

2 bottles of| beer on the wall. 

2 bottles of beer on the wall. 

2 bottles of beer. 

Take one down. 

Pass it around. 

1 bottle of beer on the wall. 

1 bottle of beer on the wall. 

1 bottle of beer. 

Take one down. 

Pass it around. 

No more bottles of beer on the wall 


»> 


Ln: 660 


Coi: 12 


38 Chapter 1 
























the basies 


Python Code Is Easy to Read 


O 







There isn’t one! 

When most programmers new to Python 
first encounter code like that of the beer 
song, they assume that something 5 s got to 
give somewhere else. 

There has to be a catch, doesn’t there? 

No, there doesn’t. It’s not by accident that 
Python code is easy to read: the language 
was designed with that specific goal in mind. 
Guido van Rossum , the language’s creator, 
wanted to create a powerful programming 
tool that produced code that was easy to 
maintain, which meant code created in 
Python has to be easy to read, too. 
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losing your mind? 


1$ Indentation Priving You Crazv? 




Indentation takes time to get used to. 

Don’t worry. Everyone coming to Python from a “curly- 
braced language” struggles with indentation at first. But it 
does get better. After a day or two of working with Python, 
you’11 hardly notice you’re indenting your suites. 


One problem that some programmers do have with 
indentation occurs when they mix tabs with spaces. Due to 
the way the interpreter counts whitespace, this can lead 
to problems, in that the code “looks fine” but refuses to run. 
This is frustrating when you’re starting out with Python. 

Our advice: dond mix tabs with spaces in your Python code. 

In fact, we’d go even further and advise you to conflgure 
your editor to replace a tap of the Tab key with four spaces 
(and while you’re at it, automatically remove any trailing 
whitespace, too). This is the well-established convention 
among many Python programmers, and you should 
follow it, too. We’ll have more to say about dealing with 
indentation at the end of this chapter. 



(rettiwg back to the beer song code 


If you take a look at the invocation of range in the beer song, you’11 notice 
that it takes three arguments as opposed to just one (as in our first example 
program). 

Take a closer look, and without looking at the explanation on the next page, 
see if you can work out what’s going on with this call to range: 


beersong.py - /Users/Paul/Desktop/ NewBook/chQ1,««. 


word = TT bottles" 
for beer_num in range (99, 
print (beer_num, word, 
print (beer_num, word. 


0 , - 1 ): 

TT of beer on' 
TT of beer . TT y 


«i 


This is hcw-* the £dll 
*to Vdhge" -takes 
thv-ee 3v-gumeh-ts, 
^o-t ohe* 
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Askmg the Interpreter for Help on a 
Function 


Recall that you can use the shell to ask for help with anything to do with 
Python, so let’s ask for some help with the range function. 

When you do this in IDLE, the resulting documentation is more than a 
screen 5 s worth and it quickly scrolls off the screen. All you need to do is scroll 
back in the window to where you asked the shell for help (as that’s where the 
interesting stuff about range is): 


»> help (range) 

Help on class range in module builtins: 



The irahge" -fUdiioh 
q» be 'mvoked in one 
ot buo ways. 


class range(object) 

| range(stop) -> range object 
| range(start, stop[, step]) -> range object 

I 

| Return a sequence of numbers from start to stop by step. 

'K 

TViis looks like it wll ^ 


• • • 


Startmg, stoppiwg, and steppiwg 


Yi 


bat we need beve- 


As range is not the only place youdl come across start, stop, and step, 
let 5 s take a moment to describe what each of these means, before looking at 
some representative examples (on the next page): 


o 

e 

o 


The START value lets you control from WHERE the range begins. 

So far, weVe used the single-argument version of range, which—from the documentation— 
expects a value for stop to be provided. When no other value is provided, range defaults to 
using 0 as the start value, but you can set it to a value of your choosing. When you do, you 
must provide a value for stop. In this way, range becomes a multi-argument invocation. 

The STOP value lets you control WHEN the range ends. 

WeVe already seen this in use when we invoked range (5) in our code. Note that the range 
that’s generated never contains the stop value, so it 5 s a case of up-to-but-not-including stop. 

The STEP value lets you control HOW the range is generated. 

When specifying start and stop values, you can also (optionally) specify a value for step. By 
default, the step value is 1, and this telis range to generate each value with a stride of 1; that 
is, 0, 1,2, 3, 4, and so on. You can set step to any value to adjust the stride taken. You can 
also set step to a negative value to adjust the directiori of the generated range. 


you are here ► 
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Experiwenting with Ranges 

Now that you know a little bit about start, stop, and step, let’s experiment at 
the shell to learn how we can use the range function to produce many different 
ranges of integers. 

To help see what’s going on, we use another function, list, to transform 
range’s output into a human-readable list that we can see on screen: 


»> range (5) 
range(0, 5) 


»> list (range (5) ) 
[0, 1, 2, 3, 4] <r 


TViis is hou, «e used W m ou<r ti«t ^3*. 


Feeding "the out?ut tvon> 'Vange* to 'list ?v-odudes a list- 


»> list (range (5, 10) 

[5, 6, 7, 8, 9] 

»> list (range (0, 10, 2)) 

[0, 2, 4, 6, 8] 


We dan adjust the START and STOP values -for 'Vange” 


It is also possible to adjwst the STEP value- 


>» 1 i s t ( range (10, 0 , - 2) ) _ Things get veally intev-esting when you adjust the 

[10, 8, 6, 4, 2] vange's div-edtion by negating the STEP value- 


»> list (range (10 , 0, 2)) <£— 

[] 

»> list (range (99, 0, -1)) 
[99, 98, 97, 96, 95, 94, 93, 


Python uiont sto? yo* ^ siI 1 Y' ffp^os.tive 
vale is biggev- than your STOP value, and STEP -s ?os,t e, 

you get badk nothing (in this dase, an en.?ty l*tl 


92, 


• • • 


5, 4, 3, 2, 1] 


After all of our experimentations, we arrive at a range invocation (shown last, 
above) that produces a list of values from 99 down to 1, which is exactly what 
the beer song 5 s for loop does: 


• • 


beersong.py - /Users/Paul/Desktop/. Nev/Book/chOt, 


l 


word = ” bottles" 

for beer_num in range (99, 
print (beer_num, word, 
print (beer_num, word. 


0, -1): ~- 
“of beer on^ 
"of beer.")' 




The £^1! "to 
takes thiree 
av-gumehts: sfor-fc, 

dhd s-fcep. 
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r i^tirpen your pencil 


word = "bottles" 


for beer num in range(99, 0, -1) 


Here again is the beer code, which has been spread out over the 
entire page so that you can concentrate on each line of code 
that makes up this"serious business applicationi 

Grab your pencil and, in the spaces provided, write in what you 
thought each line of code does. Be sure to attempt this yourself 
before looking at what we came up with on the next page. We've 
got you started by doing the first line of code for you. 

Assiy* *tbe value w bo*ttles w (a strinj) *bo a 
nev/ vaviablc dalled w word w - 


print(beer num, word, "of beer on the wall.") 


print(beer num, word, "of beer.") 


print("Take one down.") 


print("Pass it around.") 


if beer num == 1: 


print("No more bottles of beer on the wall.") 


else: 


new num = beer num - 1 


if new num == 1: 


word = "bottle" 


print(new num, word, "of beer on the wall.") 


print() 


you are here ► 
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beer explained 




Here again is the beer code, which has been spread out over the 
entire page so that you can concentrate on each line of code 
that makes up this"serious business applicationi 

You were to grab your pencil and then, in the spaces provided, 
write in what you thought each line of code does. We did the first 
line of code for you to get you started. 

How did you get on? Are your explanations similar to ours? 


word = "bottles" 


for beer num in range(99, 0, -1): 


print(beer num, word, "of beer on the 


print(beer num, word, "of beer.") 


print("Take one down.") 


print("Pass it around.") 


if beer num == 1: 


print("No more bottles of beer on 


else: 


new num = beer num - 1 


if new num == 1: 


word = "bottle" 


print(new num, word, "of beer on 


print() 


Assiy* tbe value “bottles” (a stri*<^) to a 
*ew variabie dalled “word”. 

Loop a spedi-fied *umber o-f “times, -from 

°f\ dovm to *o*e. Use u beev*_as tbe 

wal1 • loop iteratio* variabie. 

Tbe -four dalls to tbe pri*t -fu*dtio* 
display tbe durre*t iteratio*^ so*^ 
lyrids, bottles of beer o* tbe v/all* °[°[ 
bottles o-f beer. Take o*e dovm. Pass it 
arou*d”, a*d so o* v/itb eadb iteratio*. 

Cbedk to see i-f we are o* tbe last 
passed-arou*d beer... 

the wall ") we aire ' the son^ lyvids. 

Otberwise... 

Remember tbe *umber o-f tbe *e*t beer i* 
a*otber variabie dalled U *eu/__*um”. 

|-f v/eVe about to dri*k our last beer... 

Cba*ge tbe value o-f tbe “word” variabie 
so tbe last li*es o-f tbe lyrie r*ake se*se. 

Co replete tbis iteratio*^ so*^ lyrids. 

the wall.") . 

At tbe e*d o-f tbis iteratio*, pri*t a 
bla*k li*e- I/Vbe* all tbe iteratio*s are 
domplete, termi*ate tbe probram. 
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Pow't Forget to Try the Peer Sowg Code 


If you haven’t done so already, type the beer song code into IDLE, save it as beersong . py, 
and then press F5 to take it for a spin. Do not move on to the next chapter untilyou have a working beer 
song. 

^tlierejUre no 

Dtimb Questions 


M.- I keep getting errors when I try to run my beer 
song code. But my code looks fine to me, so l’m a little 
frustrated. Any suggestions? 

The first thing to check is that you have your 
indentation right. If you do, then check to see if you have 
mixed tabs with spaces in your code. Remember: the code 
will look fine (to you), but the interpreter refuses to run it. If 
you suspect this, a quick fix is to bring your code into an 
IDLE edit window, then choose Edit...-*Select AII from the 
menu system, before choosing Format...-*Untabify Region 
If youVe mixed tabs with spaces, this will convert all your 
tabs to spaces in one go (and fix any indentation issues). 


You can then save your code and press F5 to try running it 
again. If it stili refuses to run, check that your code is exactly 
the same as we presented in this chapter. Be very careful 
of any spelling mistakes you may have made with your 
variable names. 


Q.: The Python interpreter won’t warn me if I misspell 

new numasnwe num? 


No, it won’t. As long as a variable is assigned a value, 
Python assumes you know what you’re doing, and continues 
to exeeute your code. It is something to watch for, though, 
so be vigilant. 



Wrappiwg up what you already know 

Here are some new things you learned as a resuit of working through (and 
running) the beer song code: 



BULLET POINTS 


■ Indentation takes a little time to get used to. Every 
programmer new to Python complains about 
indentation at some point, but don't worry: soon 
youll not even notice you're doing it. 


The range function can take more than one 
argument when invoked. These arguments let you 
control the start and stop values of the generated 
range, as well as the step value. 


■ If there’s one thing that you should never, ever 
do, ifs mix tabs with spaces when indenting 
your Python code. Save yourself some future 
heartache, and don’t do this. 


The range function’s step value can also be 
specified with a negative value, which changes the 
direction of the generated range 


With all the beer gone, whafs next? 


That’s it for Chapter 1. In the next chapter, you are going to learn a bit more 
about how Python handles data. We only just touched on lists in this chapter, 
and ifs time to dive in a little deeper. 
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Chapter ls Code 


from datetime import datetime 


odds = [ 1, 

3, 

5, 

7, 

9, 

11, 

13, 

15, 

17, 

19, 

21, 

23, 

25, 

27, 

29, 

31, 

33, 

35, 

37, 

39, 

41, 

43, 

45, 

47, 

49, 

51, 

53, 

55, 

57, 

59 ] 


right this minute = datetime.today().minute 


We sia<ried wi-fch 

the "oddpy" 
probam, iheh... 


if right_this_minute in odds: 

print( M This minute seems a little odd. M ) 


else: 

print( M Not an odd minute.") 


... e^tended the CoAt to 


dv-eate w odd2-py”, 'whidh v-a« 
■the Vmute dhedkmg dode" 
Jpive (thanks t o the use 


0 -f Python s "-for” loop)- 




from datetime import datetime 

import random 
import time 


odds = [ 1, 

3, 

5, 

7, 

9, 

11, 

13, 

15, 

17, 

19, 

21, 

23, 

25, 

27, 

29, 

31, 

33, 

35, 

37, 

39, 

41, 

43, 

45, 

47, 

49, 

51, 

53, 

55, 

57, 

59 ] 


for i in range(5): 

right_this_minute = datetime.today().minute 
if right_this_minute in odds: 

print( M This minute seems a little odd. M ) 
else: 

print( M Not an odd minute.") 
wait_time = random.randint(1, 60) 

time.sleep(wait time) 


word = "bottles" 





for beer num in range(99, 0, - 

1) : 



print(beer num. 

word, "of 

beer on 

the 

wall.") 

print(beer num. 

word, "of 

beer.") 



print("Take one 

down.") 




print ("Pass it 

around.") 




if beer num == 

1: 




print("No more bottles 

of beer 

on 

the wall.") 

else: 





new num = beer num - 1 




if new num 

== 1: 




word = 

"bottle" 




print(new num, word, " 

of beer 

on 

the wall.") 

print () 







Wt -this 

dha tf» with the Pytho* 
vcirsio* O-P the ttedd 

Fiv-s-fc ^Iassie u bcc\T so*g. w 
Ahd ; yes, we k*ow: it s 
ba\rd hot to work o* 

this £ode without si*ai*a 

alo*g... © 
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* Working with Ordered Data + 



AII programs process data, and Python programs are no exception. 

In fact, take a look around: data is everywhere. A lot of, if not most, programming is all about 
data: acquiring data, Processing data, understanding data. To work with data effectively, you need 
somewhere to put your data when Processing it. Python shines in this regard, thanks (in no small 
part) to its inclusion of a handful of widely applicable data structures: lists, dictionaries, tuples, and 
sets. In this chapter, we’ll preview all four, before spending the majority of this chapter digging deeper 
into lists (and we’ll deep-dive into the other three in the next chapter). We’re covering these data 
structures early, as most of what you’ll likely do with Python will revolve around working with data. 


this is a new chapter 
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Nuwbers, Strmg$...and Objccts 

Working with a single data value in Python works just like you’d expect it to. 
Assign a value to a variable, and you’re all set. With help from the shell, let 5 s 
look at some examples to recall what we learned in the last chapter. 


Nuwbers 

Let 5 s assume that this example has already imported the random module. 

We then call the random. randint function to generate a random number 
between 1 and 60, which is then assigned to the wait_time variable. As 
the generated number is an integer, that’s what type wait_time is in this 
instance: 

»> wait_time = random. randint (1, 60) 
»> wait_time 
26 

Note how you didn’t have to teli the interpreter that wait_time is going to 
contain an integer. We assigned an integer to the variable, and the interpreter 
took care of the details (note: not all programming languages work this way). 

Striwgs 

If you assign a string to a variable, the same thing happens: the interpreter 
takes care of the details. Again, we do not need to declare ahead of time that 
the word variable in this example is going to contain a string: 

»> word = "bottles" 

»> word 
1 bottles 1 

This ability to dynamically assign a value to a variable is Central to Python’s 
notion of variables and type. In fact, things are more general than this in that 
you can assign anything to a variable in Python. 

Objects 

In Python everything is an object. The means that numbers, strings, functions, 
modules— everything —is an object. A direct consequence of this is that all 
objects can be assigned to variables. This has some interesting ramifications, 
which wedl start learning about on the next page. 


A variahle 
takes on the 
type ol the 
value assigned. 


Everything is an 
ohject in Python 
and any ohject 
can he assigned 
to a variahle. 
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"Everything Is an Object" 


Any object can be dynamically assigned to any variable in Python. Which begs the 
question: whafs an object in Python? The answer: everything is an object. 

All data values in Python are objects, even though—on the face of things—“Don’t 
panic!” is a string and 42 is a number. To Python programmers, “Don’t panic!” is a 
string object and 42 is a number object. Like in other programming languages, objects can 
have state (attributes or values) and behavior (methods). 






Sort of. 

You can certainly program Python in an object-oriented way 
using classes, objects, instances, and so on (more on all of this later 
in this book), but you don’t have to. Recall the programs from the 
last chapter...none of them needed classes. Those programs just 
contained code, and they worked fine. 

Unlike some other programming languages (most notably, Java), 
you do not need to start with a class when first creating code in 
Python: you just write the code you need. 


Now, having said all that (and just to keep you on your toes), 
everything in Python behaves as if it is an object derivedfrom some 
class. In this way, you can think of Python as being more object- 
based as opposed to purely object-oriented, which means that 
object-oriented programming is optional in Python. 


Put...what does all this actually meaw? 

As everything is an object in Python, any “thing” can be assigned to any variable, and 
variables can be assigned any thing (regardless of what the thing is: a number, a string, a 
function, a widget...any object). Tuck this away in the back of your brain for now; we’11 
retura to this theme many times throughout this book. 

There 5 s really not a lot more to storing single data values in variables. Let’s now take a 
look at Python’s built-in support for storing a collection of values. 


you are here ► 
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Meet tbe Four Puilt-in Pata Structores 


Python comes with four built-in data structures that you can use to hold any 
collectiori of objects, and they are list, tuple, dictionary, and set. 

Note that by “built-in” we mean that lists, tuples, dictionaries, and sets are always 
available to your code and they do not need to be importedprior to use: each of these 
data structures is part of the language. 

Over the next few pages, we present an overview of all four of these built-in data 
structures. You may be tempted to skip over this overview, but please don’t. 

If you think you have a pretty good idea what a list is, think again. Python’s list 
is more similar to what you might think of as an array, as opposed to a linked-list, 
which is what often comes to mind when programmers hear the word “list.” (If 
you’re lucky enough not to know what a linked-list is, sit back and be thankful). 

Python’s list is the first of two ordered-collection data structures: 



List: an ordered mutable collection of objects 

A list in Python is very similar to the notion of an array in other 
programming languages, in that you can think of a list as being an indexed 
collection of related objects, with each slot in the list numbered from zero 
upward. 

Unlike arrays in a lot of other programming languages, though, lists are 
dynamic in Python, in that they can grow (and shrink) on demand. There 
is no need to predeclare the size of a list prior to using it to sto re any objects. 

Lists are also heterogeneous, in that you do not need to predeclare the type 
of the object you’re storing—you can mix’n’match objects of different types 
in the one list if you like. 


Lists are mutable, in that you can change a list at any time by adding, 
removing, or changing objects. 


Lists da* 
dynamidally storink 
and yro* to any 


SJZjC- 


01ojctk are stored 
in individual slots 
in the list 



( 

object 


( 

< 

object 


( 

object 


( 

object 


S, 

object 



4 

3 

2 

1 

0 


As With arrays, slot; 

are numbered -from 

«ro upy/ard...these 

are “index values." 



List 


A list is like 

an array— 
tne objects 
it stores 
are ordered 
secjuentially 
in slots. 
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Ordered Collectioris Are Mutable/lwwutable 

Python’s list is an example of a mutable data structure, in that it can change (or 
mutate) at runtime. You can grow and shrink a list by adding and removing objects as 
needed. It’s also possible to change any object stored in any slot. We’ll have lots more 
to say about lists in a few pages’ time as the remainder of this chapter is devoted to 
providing a comprehensive introduction to using lists. 

When an ordered list-like collection is immutable (that is, it cannot change), it’s 
called a tuple: 



Tuple: an ordered immutable collection of objects 

A tuple is an immutable list. This means that once you assign objects to a tuple, 
the tuple cannot be changed under any circumstance. 

It is often useful to think of a tuple as a constant list. 

Most new Python programmers scratch their head in bemusement when they 
first encounter tuples, as it can be hard to work out their purpose. After ali, 
what use is a list that cannot change? It turns out that there are plenty of use 
cases where you’ll want to ensure that your objects can’t be changed by your (or 
anyone else’s) code. We’ll return to tuples in the next chapter (as well as later in 
this book) when we talk about them in a bit more detail, as well as use them. 


Tuples ave like lists, 
c%£cpt ofiCt eveated 

thev CANNOT 
ehan^e- Tuples ave 
tonstant lists- 


\ 

object 


( 

object 


( 

object 



Tuple 


2 

1 

0 

^ Tuples use mde* 
^— values, too (just 
like lists)- 


Lists and tuples are great when you want to present data in an ordered way (such as a 
list of destinations on a travel itinerary, where the order of destinations is important). 
But sometimes the order in which you present the data isrdt important. For instance, 
you might want to store some user’s details (such as their id and password), but you 
may not care in what order they’re stored (just that they are). With data like this, an 
alternative to Python’s list/tuple is needed. 



is an 

immutable 

list. 
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Aw Unordered Pata Structure: Pictiowary 

If keeping your data in a specific order isn’t important to you, but structure 
is, Python comes with a choice of two unordered data structures: dictionary 
and set. Let 5 s look at each in turn, starting with Python 5 s dictionary 


Depending on your programming background, you may already know what a 
dictionary is, but you may know it by another name, such as associative array, 
map, Symbol table, or hash. 

Like those other data structures in those other languages, Python’s dictionary 
allows you to sto re a collection of key/value pairs. Each unique key has a value 
associated with it in the dictionary, and dictionaries can have any number of 
pairs. The values associated with a key can be any object. 

Dictionaries are unordered and mutable. It can be useful to think of Python’s 
dictionary as a two-columned, multirow data structure. Like lists, dictionaries 
can grow (and shrink) on demand. 


© Dictionary: an unordered set of key/value pairs 


pittionavies associate keys 
iwith values, a»d (like lists) ea* 

amieally skvmk a*d youi to 


a»y siz-e 


• • 

• • 


key#4 


object 

key#l 

i 

object 

key#3 

i 

object 

key#2 

i 

object 


Dictionary 





A dictionary 
stores key/ 
value pairs. 


Something to watch out for when using a dictionary is that you cannot rely 
upon the internal ordering used by the interpreter. Specifically, the order 
in which you add key/value pairs to a dictionary is not maintained by the 
interpreter, and has no meaning (to Python). This can stump programmers 
when they first encounter it, so we’re making you aware of it now so that 
when we meet it again—and in detail—in the next chapter, you’11 get less of a 
shock. Rest assured: it is possible to display your dictionary data in a specific 
order if need be, and we’ll show you how to do that in the next chapter, too. 
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A Pata Structure That Avoids Puplicates: Set 

The final built-in data structure is the set, which is great to have at hand when you want 
to remove duplicates quickly from any other collection. And don’t worry if the mention 
of sets has you recalling high school math class and breaking out in a cold sweat. 
Python’s implementation of sets can be used in lots of places. 



Set: an unordered set of unique objects 

In Python, a set is a handy data structure for remembering a collection of 
related objects while ensuring none of the objects are duplicated. 

The fact that sets let you perform unions, intersections, and differences is an 
added bonus (especially if you are a math type who loves set theory). 


Sets, like lists and dictionaries, can grow (and shrink) as needed. Like dictionaries, 
sets are unordered, so you cannot make assumptions about the order of the 
objects in your set. As with tuples and dictionaries, you’11 get to see sets in action 
in the next chapter. 


TWmk olr 3 sex- 

as a tollefction ok 

wnovdeved 


4 • A 

allowed- 


objecta 0b j ecte 
objectf 

object c ° bjeCtV 


A set does not 
allow duplicate 
objects. 


Set 


The $0/20 data structure rule of thuwb 

The four built-in data structures are useful, but they don’t cover every possible data 
need. However, they do cover a lot of them. It’s the usual story with technologies 
designed to be generally useful: about 80% of what you need to do is covered, while 
the other, highly specific, 20% requires you to do more work. Later in this book, you’11 
learn how to extend Python to support any bespoke data requirements you may 
have. However, for now, in the remainder of this chapter and the next, we’re going to 
concentrate on the 80% of your data needs. 

The rest of this chapter is dedicated to exploring how to work with the first of our four 
built-in data structures: the list. We’ll get to know the remaining three data structures, 
dictionary, set, and tuple, in the next chapter. 


you are here ► 


53 















lists are everywhere 


A List Is an Ordered Collection of Objects 

When you have a bunch of related objects and you need to put them somewhere 
in your code, think list. For instance, imagine you have a montffs worth of daily 
temperature readings; storing these readings in a list makes perfect sense. 

Whereas arrays tend to be homogeneous affairs in other programming languages, 
in that you can have an array of integers, or an array of strings, or an array of 
temperature readings, Python’s list is less restrictive. You can have a list of objects , 
and each object can be of a differing type. In addition to being heterogeneous, 
lists are dynamic: they can grow and shrink as needed. 

Before learning how to work with lists, let 5 s spend some time learning how to spot 
lists in Python code. 


object 


object 


object 


object 


object 


4 

3 

2 

1 

0 


List 


How to spot a list in code 


Lists are always enclosed in square brackets, and the objects contained within 
the list are always separated by a comma. 

Recall the odds list from the last chapter, which contained the odd numbers from 
0 through 60, as follows: 


The list stavts with a* 
ofen'm$ s<\uave lovadket- 


odds = 


[ 1, 

3, 

5, 

7, 

9, 

11, 

13, 

15, 

17, 

19, 

21, 

23, 

25, 

27, 

29, 

31, 

33, 

35, 

37, 

39, 


43, 

45, 

47, 

49, 

51, 

53, 

55, 

57, 

59 


~V 

oh 


V The data values (a k a- "the 
V objetts") ave sepavated t 


Yonr\ oir\C 


A list ot the 
odd humbev-s 


rhe list ehds with a 
<llosmg s^uave bvadket- 


When a list is created where the objects are assigned to a new list directly in 
your code (as shown above), Python programmers refer to this as a literal 
list, in that the list is created and populated in one go. 

The other way to create and populate a list is to “grow” the list in code, 
appending objects to the list as the code executes. We’ll see an example of 
this method later in this chapter. 

Let’s look at some literal list examples. 


Lists can te 
created literally 


or "gfrown” in c 


ode. 
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data 


Creating Usts Literally 

Our first example creates an empty list by assigning [ ] to a variable called prices: 




4 

3 

2 

1 

0 


List 


Here 5 s a list of temperatures in degrees Fahrenheit, which is a list of floats: 



How about a list of the most famous words in computer programming? Here they are: 


Objeets C\t\ this case, 
Some floats) are 
separa-fced by eommas 
3hd su\r\rouhdcd by 
b\raekets-it's 

a list 



A list of 
sfarinj objects 


Here’s a list of car details. Note how it is OK to sto re data of mixed types in a list. 
Recall that a list is “a collection of related objects.” The two strings, one float, and one 
integer in this example are ali Python objects, so they can be stored in a list if needed: 



Our two final examples of literal lists exploit the fact that—as in the last example— 
everything is an object in Python. Like strings, floats, and integers, lists are objects, too. 
Here’s an example of a list of list objects: 


di-f-Pev-ma 

ty pe 



And here’s an example of a literal list of literal lists: 



odds_and_ends = [ [ 1, 2, 3], ['a', ' b' , 'c' ], 

[ 'One ', ' Two ', ' Three' ] ] 



Do*/t wo\rvy i-f 
these last two 
examples are 
-(reakmg you 
out* I/Ve wo*/t 
be wovkihg with 
a^ythihg as domplex 
as this uhtil a latcv- 
£hapter. 


you are here ► 
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lists at work 


Puttiwg Lists to Work 

The literal lists on the last page demonstrate how quickly lists can be created 
and populated in code. Type in the data, and you’re off and running. 

In a page or two, we’ll cover the mechanism that allows you to grow (or 
shrink) a list while your program executes. After ali, there are many situations 
where you don’t know ahead of time what data you need to store, nor how 
many objects you’re going to need. In this case, your code has to grow (or 
“generate”) the list as needed. You’11 learn how to do that in a few pages’ time. 

For now, imagine you have a requirement to determine whether a given 
word contains any of the vowels (that is, the letters a , e, i , o , or u). Can we use 
Python’s list to help code up a solution to this problem? Let’s see whether we 
can come up with a solution by experimenting at the shell. 



4 

3 

2 

1 

0 


List 


Workiwg with lists 


We’ll use the shell to first define a list called vowels, then check to see if 
each letter in a word is in the vowels list. Let’s define a list of vowels: 

»> vowels = [' a' , ' e' , ' i' , ' o' , f u f ] 


h list o-f -the 
-five vowels 


With vowels defined, we now need a word to check, so let’s create a 
variable called word and set it to M Milliways": 


Heve’s a 'wov-d 
to thedk- 



»> word = 


"Milliways" 


Is one object inside another? Check with "in" 

If you remember the programs from Ghapter 1, you will recall that we 
used Python’s in operator to check for membership when we needed to ask 
whether one object was inside another. We can take advantage of in again 
here: 



We're only using the letters 
oeiou as vowels, even though 
the letter y is considered to be 
both a vowel and a consonant. 


»> for letter in word: 


if letter in vowels: 
print(letter) 

V 



lakc eadh letter in the wovd... 
...and i-f it is in 'the u vowel s!* list 
_ ...disflay letter on s^een. 


i 

a 


The outfut tv-om this tode the ide^tity 

the vowels m the wovd "/Williways". 


Let’s use this code as the basis for our working with lists 
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data 


Use Your Editor When Workiwg ow More 
Than a Few lines of Code 

In order to learn a bit more about how lists work, let 5 s take this code and 
extend it to display each found vowel only once. At the moment, the code 
displays each vowel more than once on output if the word being searched 
contains more than one instance of the vowel. 

First, let 5 s copy and paste the code youVe just typed from the shell into a new 
IDLE edit window (select File... —► New File... from IDLE’s menu). We’re going 
to be making a series of changes to this code, so moving it into the editor 
makes perfect sense. As a general rule, when the code we’re experimenting 
with at the >>> prompt starts to run to more than a few lines, we find it more 
convenient to use the editor. Save your five lines of code as vowels . py. 

When copying code from the shell into the editor, be careful not to include 
the >>> prompt in the copy, as your code won’t run if you do (the interpreter 
will throw a syntax error when it encounters >>>). 

When youVe copied your code and saved your file, your IDLE edit window 
should look like this: 

# ■’> vowels.py - /LJsers/Paul/Desktop/ NawBook/ch02/vowels.py (3.4.3) 


vowels = ['a 1 , 'e', 'i‘ f 'o' f 'u 1 ] 

word = "Milliways" 
for letter in word: 

if letter in vowels: 
print (letter) 


Ln: 7 Coi: 0 


Potft forget: prcss F5 to run your prograw 

With the code in the edit window, press F5 and then watch as IDLE jumps to 
a restarted shell window, then displays the program’s output: 


% # # Python 3.4.3 Shell 



Your list e%ar*ple Lo&t 
saved as w vowels.py w mside 
ar> IPLE edit wmdow. 



object 


object 


object 


object 


object 


4 


0 


List 












































one at a time 


re 


frrowmg" a List at Runtime 


Our current program displays each found vowel on screen, including any 
duplicates found. In order to list each unique vowel found (and avoid displaying 
duplicates), we need to remember any unique vowels that we find, before 
displaying them on screen. To do this, we need to use a second data structure. 

We can’t use the existing vowels list because it exists to let us quickly determine 
whether the letter we’re currently processing is a vowel. We need a second list that 
starts out empty, as we’re going to populate it at runtime with any vowels we find. 

As we did in the last chapter, let’s experiment at the shell before making any 
changes to our program code. To create a new, empty list, decide on a new 
variable name, then assign an empty list to it. Let 5 s call our second list found. 
Here we assign an empty list ([ ]) to found, then use Python’s built-in function 
len to check how many objects are in a collection: 


object 


object 


object 


object 


object 


4 


0 


List 


»> found = [] 
»> len (found) 
0 



piy list.. 


...whith ihe 'mte*-pvetev (thanks 
■b "len”) to«Ws has no objedts. 


The "len" huilt- 
in function 
reports on the 
size ol an ohject. 


Lists come with a collection of built-in methods that you can use to manipulate 
the list’s objects. To invoke a method use the dot-notation syntax : postfix the list’s 
name with a dot and the method invocation. We’ll meet more methods later in 
this chapter. For now, let’s use the append method to add an object to the end of 
the empty list we just created: 

Add io an exisiin<j Ii si ai \ruirfcime 


»> found. append (' a') 
»> len (found) 

1 ^- 

»> found ^- 

['a'] 


usihj ihe “append* meihod. 

The lengih ihe lisi has now indveased- 

_ihe Shell to disf-lay the dontents ot the list 

tonfivms the olojedt is »onn f3<rt of the list- 


Repeated calls to the append method add more objects onto the end of the list: 


~\ 


»> found. append (' e ') 

»> found. append (' i ') V ^ 
»> found. append (' o') 

»> len (found) 

4 

»> found 

['a', 'e', 'i' 'o'] 


/Wolre iruntime 
additions 


Lists come with a 
hunch ol huilt-in 
methods. 


X 


0n£e again, we use the shell io 
don-fivr* ali is ih oirdeV". 




Let’s now look at what 5 s involved in checking whether a list contains an object. 
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data 


Checking for Membership with "in" 

We already know how to do this. Recall the “Millyways” example from a few 
pages ago, as well as the odds . py code from the previous chapter, which 
checked to see whether a calculated minute value was in the odds list: 


The w m * opevatov- 
thetks -fov- 
r«er*bevship. 



if right_this_minute in odds: 
print("This minute seems a 


little 


odd.") 


object 


object 


object 


object 


object 


4 

3 

2 

1 

0 


List 


Is the object "in" or "not Iu"? 

As well as using the in operator to check whether an object is contained 
within a collection, it is also possible to check whether an object does not exist 
within a collection using the not in operator combination. 

Using not in allows you to append to an existing list only when you know 
that the object to be added isn’t already part of the list: 


»> if 'u 


not in found: 
found.append( 1 u 1 ) 


»> found 

['a', 'e', 'i' 'o', 1 u 1 ] 

»> 

»> if ! u f not in found: 

found.append( 1 u 1 ) 


This -Piv-s-t ihVoeatioh o-f u appChd" 
wo\rks> as u does Ko-fc duwehtly 
exis-t y/rthih the ^ouhd” list fas 
you sav/ Oh the previous pacte, the 
list .Ohtaihed ck V, V, ik 

This *ext 'mvodatioh o-f u apper\d 
does hot exe£ute, as u alv-eady 
exists ih w< (’ouhd so does hot heed 
to be added a^aih- 


»> found 

['a', 'e', 'i' 'o', 1 u 1 ] 




Good catch. A set might be better here. 

But, we’re going to hold off on using a set until the next 
chapter. We’ll return to this example when we do. For now, 
concentrate on learning how a list can be generated at 
runtime with the append method. 


you are here ► 
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unique vowels only 


Itt Time to Update Our Code 

Now that we know about not in and append, we can change our code with 
some confidence. Here’s the original code from vowels . py again: 



vowels = ['a', 'e', 'i', ' o', ' u'] 

word = "Milliways" 
for letter in word: 

if letter in vowels: 
print(letter) 



Save a copy of this code as vowels2 . py so that we can make our changes to 
this new version while leaving the original code intact. 

We need to add in the creation of an empty f ound list. Then we need some extra 
code to populate f ound at runtime. As we no longer display the found vowels as 
we find them, another for loop is required to process the letters in found, and 
this second for loop needs to exeeute after the first loop (note how the indentation 
of both loops is aligned below). The new code you need is highlighted: 



4 

3 

2 

1 

0 


List 


This Codt displays 
the vowels ‘m u wovd” 
3s -fchey ave -Pouhd- 


This is 

u voweU2~py • ^ 


Stavt with 
a» empty list 



vowels = ['a', 


' e' 


'i' 


' o' 


'u' ] 


word = "Milliways" 

found = [] 

for letter in word: 

if letter in vowels: 

if letter not in found: 
found.append(letter) 
for vowel in found: 
print(vowel) 


My this w *y | oop iy 

, 9 et to <ruh, a*d it displays tfc 
vowels &u»d i» 'Wd w 7 


Include -the Code that 
detides whetheir to 
wpdate the list ot 
-roKhd vowels. 


Let’s make a final tweak to this code to change the line that sets word to 
“Milliways” to be more generic and more interactive. 

Ghanging the line of code that reads: 

word = "Milliways" 
to: 


word = input("Provide a word to search for vowels: 


") 



instructs the interpreter to prompt your user for a word to search for vowels. The 
input function is another piece of built-in goodness provided by Python. 


Make the change as suggested 
on the left, then save your 
updated code as vowe 1 s 3 . py. 
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data 



With the change at the bottom of the last page applied, and this latest version of your program 
saved as vowels3 .py, let’s take this program for a few spins within IDLE. Remember: to run your 
program multiple times, you need to return to the IDLE edit window before pressing the F5 key. 


HevVs ouv versior\ 
<Jc u vo>wels3.py 

v/ith the infut 
edit applied- 



A*d here are our 
test ruhs... 


# 0 O vowelsS.py - /Users/Paul/Desktop/ N ewBook/c h 02/vowelsS, py (3.4.3) 

vowels = [“a 1 , 'e 1 , ‘i 1 , 'o', ‘u*] 

word = input ( "Provide a word to search for vowels: 11 ) 

f ound = [] 

for letter in word: 

if letter in vowels: 

if letter not in found: 
foTind, append (letter) 
for vowel in found: 
print (vowel) 



Ln: 11 

Coi: 0 



# # # Python 3.4.3 Shell 

»> ================================ RESTART 

»> 


Provide a word to search for vowels: Milliways 
• 

i 

a 

»> ================================ RESTART ================================ 

»> 

Provide a word to search for vowels: Hitch-hiker 
• 

i 

e 

»> ================================ RESTART ================================ 

»> 

Provide a word to search for vowels: Galaxy 
a 

»> ================================ RESTART ================================ 

»> 

Provide a word to search for vowels: Sky 

»> I 

Ln: 21 Coi: 4 


Our output confirms that this small program is working as expected, and it even does the right thing 
when the word contains no vowels. How did you get on when you ran your program in IDLE? 


you are here ► 
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manipulating lists 


Rewoviwg Objects frow a List 

Lists in Python are just like arrays in other languages, and then some. 

The fact that lists can grow dynamically when more space is needed (thanks 
to the append method) is a huge productivity boon. Like a lot of other 
things in Python, the interpreter takes care of the details for you. If the list 
needs more memory, the interpreter dynamically allocates as much memory 
as needed. Likewise, when a list shrinks, the interpreter dynamically reclaims 
memory no longer needed by the list. 

Other methods exist to help you manipulate lists. Over the next four pages 
we introduce four of the most useful methods: remove, pop, extend, and 
insert: 



4 

3 

2 

1 

0 


List 



remove: tokes an objecfs value as its sole argument 

The remove method removes the first occurrence of a specified data value from a list. If 
the data value is found in the list, the object that contains it is removed from the list (and 
the list shrinks in size by one). If the data value is not in the list, the interpreter will raise an 
error (more on this later): 


»> nums 
»> nums 


[ 1 , 2 , 3 , 







This is v/hat “the 
“nums” list looks like 
be-Cove the tali 
to tbe 'Vemove 
method- 





»> nums . remove (3) 
»> nums 


This is -#Kot^ ah ihdex value; its 
the value to iremove. 


[ 1 , 2 , 4 ] 


A^tev- the tali 
to 'Vejove > the 
objett v/ith 3 as 
its value is y>he. 
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Popping Objects Off a List 

The remove method is great for when you know the value of the object you 
want to remove. But often it is the case that you want to remove an object 
from a specific index slot. 

For this, Python provides the pop method: 


The pop method removes and returns an object from an existing list based on the 
objecPs index value. If you invoke pop without specifying an index value, the last 
object in the list is removed and returned. If you specify an index value, the object 
in that location is removed and returned. If a list is empty or you invoke pop with 
a nonexistent index value, the interpreter rais es an error (more on this later). 


G pop: takes an optional index value as its argument 
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3 

2 

1 
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List 


Objects returned by pop can be assigned to a variable if you so wish, in which case 
they are retained. However, if the popped object is not assigned to a variable, its 
memory is reclaimed and the object disappears. 


Bc-fovc w f°P ,s 
we ha ve a list with 
thvee objects- 



/«* didn't teli >op" 
v/hieh ite». to lrentove, 
so it opeirates on the 
last ite» in the list- 


4 


»> nums 

[ 1 , 2 ] 



object, which is veclaimed- 


/\ttev the 
"p°p", the list 
shvihks- 




As be-Cov-e, w f°P 
vetuVhS the vemoved 
object- Otitt 
the objett is 
vetlaimed by the — 
ihtev-pveteir. 



This is ah ihdex value. Zevo 
^vespohds to the Usi objeet 
»> nums .pop (0) ,h l ,s t (the hur*ber /). 

1 


At this foiht, W hums" 

has beeh veduted to ^ 

a sih^le-item list- 


»> nums 

[ 2 ] 



The W hur*s W list has 
shlruhk to a sihale- 
item list. 



you are here ► 
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growing your list 


Extending a List with Objects 

You already know that append can be used to add a single object to an existing 
list. Other methods can dynamically add data to a list, too: 



extend: takes a list of objects as its sole argument 

The extend method takes a second list and adds each of its objects to an existing 
list. This method is very useful for combining two lists into one: 



4 

3 

2 

1 

0 


List 


This is what 
the list 
duvv-eirtly looks like : 
it is d sm^le-ite** 

list 





»> nums . extend ([ 3, 4 ]) 
[2 r 3 f 4] 


Pavide a lis-t <4 

objctis io append 
w> the existing list. 


Me* ve e%ter\ded “this \[av*s 
list by takin^ eadh o( the 
objedts m the pvovided list 
a*d appendi*^ iis objedts. 




i C 

»> nums . extend ([ ]) 
[2, 3, 4] 


w veii you re 
adding no items to the end ot 

existing list). It you'd instead 
«lled appendO", an en, P ty list 
wouId be added to the end ot the 
existing list, but-in this exa«,p| e _ 
USIM 9 «aendCj)" does nothing. 


Bedause the er*pty list used to -^ 

extend the w nums* list £oivtamed 
no objedts, rothin^ dban^cs. 
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Iwsertmg an Object into a List 

The append and extend methods get a lot of use, but they are restricted to 
adding objects onto the end (the righthand side) of an existing list. Sometimes, 
youil want to add to the beginning (the lefthand side) of a list. When this is the 
case, you’11 want to use the insert method. 



insert: takes an index value and an object as its arguments 

The insert method inserts an object into an existing list before a specified index 
value. This lets you insert the object at the start of an existing list or anywhere 
within the list. It is not possible to insert at the end of the list, as that’s what the 
append method does: 



4 
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2 

1 
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List 


IW. k» w' wi»** 

a-f-tev ali tha-t e*tend>n3 *' r0 "' th 

py-evious 


»> nums . insert (0 , 1) 

»> nums 
[1, 2, 3, 4] 

The inde* of *the object 
*to insert ^be-Pov-e^ 


T X— The value (aka "objefrt") 


■fco insert 


hi 




Baek -feo y/heve y/e s-tavted 


After all that removing, popping, extending, and inserting, weVe ended up with the 
same list we started with a few pages ago: [ 1 , 2 , 3 , 4 ]. 


Note how it’s also possible to use insert to add an object into any slot in an 
existing list. In the example above, we decided to add an object (the number 1) to 
the start of the list, but we could just as easily have used any slot number to insert 
into the list. Let’s look at one final example, which—just for fun—adds a string into 
the middle of the nums list, thanks to the use of the value 2 as the first argument 



And there i-fc is—-the 

■final "nums" list, u/hieh 
has -five objed-ts: -fouv 
nurwbevs and ohe stvmg. 


Let’s now gain some experience using these list methods. 


you are here > 
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just like arrays? 


What About Usiwg Square Prackcts? 



I'm a little confused. You keep telling me that 
lists are "just like arrays in other programming 
languages," but youve yet to say anything about the 
square bracket notation I use with arrays in my other 
favorite programming language. What gives? 


Don’t worry, we’re going to get to that in a bit. 

The familiar square bracket notation that you know and love 
from working with arrays in other programming languages 
does indeed work with Python’s lists. However, before we get 
around to discussing how, let 5 s have a bit of fun with some of 
the list methods that you now know about. 


tliere.are no 

- Dumb Qiiestions - 

How do I find out more about these and any other list methods? 

You ask for help. At the »> prompt, type help (list) to access Pythorfs list documentation (which provides a few pages of 
material) or type help (list. append) to request just the documentation for the append method. Replace append with any other 
list method name to access that method’s documentation. 
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Time for a challenge. 

Before you do anything else, take the seven lines of code shown below and 
type them into a new IDLE edit window. Save the code as panic . py, and 
execute it (by pressing F5). 

Study the messages that appear on screen. Note how the first four lines of code 
take a string (in phrase), and turn it into a list (in plist), before displaying 
both phrase and plist on screen. 


The other three lines of code take plist and transform it back into a string (in 
new phrase) before displaying plist and new phrase on screen. 


Your challenge is to transform the string "Don' t panic ! " into the string 
"on tap" using only the list methods shown thus far in this book. (There's no 
hidden meaning in the choice of these two strings: it's merely a matter of the 
letters in "on tap"appearing in "Don' t panic ! "). At the moment, panic . 
py displays "Don' t panic ! " twice. 


We ave stav-tm^ 
wth a stvin(y 


Hint: use a for loop when performing any operation multiple times. 


We turh the 
strihg ifito 3 list 


phrase = "Don't panic!" 
plist = list (phrase) 
print(phrase) 
print(plist) 






Wc display the string 
3hd the list Oh sereeh. 



Add your list 
mahifulatioh tode 
here- 


new_phrase = join (plist) 

We display the f print(plist) 

trahs£orr*ed list 3hd print (new_phrase) 

the he>w strmg oh sdrech. 



This lihe takes the 
list 3hd turhs it 
baek ihto a strihg. 


you are here ► 
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It was time for a challenge. 


017 tap 

Sharpen your pencil 

Solutiori 


You y/eve to add youv list 
manipulation dode beve- 
Tbis is wbat we daime 
u p y/itb—don t wovry it 
youvs is vevy di£Cevent 
tvom ouvs. Tbeve s 
move tban one way to 
pev-PoVm tbe nedessavy 
tv^anstoirmations usin<^ tbe 


£jet vid ot tbe 
'P' at tbe stavt' 
ot tbe list- 


Swap tbe two objcdts at 
tbe end ot tbe list ky 
-f iv-st poppin^ eadb objedt 
-Cvom tbe list) tben usin$ 
tbe poppcd objedts to 
e*tend tbe list- Tbis is a 
line ot dode tbat you II 
need to tbink akout toir a 
litti e bit &y P 0, nt : tbe 
pops odduv -^tivst^ (in 
tbe ovdev sbown)> tben 
tbe e*tend bappens. 


Before you did anything else, you were to take the seven lines of code shown 
on the previous page and type them into a new IDLE edit window, save the 
code as panic . py, and exeeute it (by pressing F5). 

Your challenge was to transform the string "Don' t panic ! " into the 
string "on tap” using only the list methods shown thus far in this book. 
Before your changes, panic . py displayed "Doni panic!" twice. 

The new string (displaying "on tap") is to be stored in the new_phrase 
variable. 



phrase = M Don't panic!" 
plist = list (phrase) 
print (phrase) 
print (plist) 


-fov i in van^e(^") : 
plistpopO 


"Tbis small loop pops tbe 
last touv obiedts -pVom 

V'st” move Vid/" 



^ plist pop(O) 

plist-vemoveO 1 l> ) 

plist e^tend(Cplist popO, plist popO]) 

plist insevt(2-, plist-pop(3>)) 

new_phrase = join(plist) 
print(plist) 
print(new phrase) 


nnd, then remove, -the apostrophe 
-rv-ottv the list- 


This Ime of eode pops the spade from the 
list, the» inserts it badk mto the list at 
index lodation Z. Just like the last Ime of 
tode, the pop oddurs *{irst*, be-fore the 
insert happens. And, remember: spades are 
dbavadtevs, too. 


As there’s a lot going on in this exercise solution, the next two 
pages explain this code in detail. 
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What Happened to "plist"? 

Let 5 s pause to consider what actually happened to plist as the code in 
panic. py executed. 

On the left of this page (and the next) is the code from panic . py, which, 
like every other Python program, is executed from top to bottom. On the 
right of this page is a visual representation of plist together with some 
notes about what’s happening. Note how plist dynamically shrinks and 
grows as the code executes: 

The Code The State of plist 


object 


object 


object 


object 


object 


List 


4 

3 

2 

1 

0 


At this point in the code, plist does not yet exist. The 
phrase = "Don ' t panic ! " second line of code transforms the phrase string into a 

new list, which is assigned to the plist variable: 


plist = list(phrase) 

- > 

■ 

D 

o 

' 

Y\ 

1 

■ 

i 


' 

F 

• 

a 

• 

Y\ 

| 

i 

i 

t 

' 

1 




_ 













0 

1 

z 

3 

4- 

5 

b 

i 

0 

i 

10 

II 


print (phrase) 1 _ ^These £alls -fco display 

print (plist) the £um*t state o-P the 

vaviables (be-Pome we stavt 

ouV" ma^ipulatiohs). 


for i in range(4): 

plist.pop() 



Each time the for loop ite rates, plist shrinks by one 
object until the last four objects are gone: 



The loop terminates, and plist has shrunk until eight 
objects remain. It’s now time to get rid of some other 
unwanted objects. Another call to pop removes the first 
item on the list (which is at index number 0): 




■ 

> 

1 

i 


' 

| 

plist.pop(0) —___^ 

o 

h 


t 


F 

a 


ol z 1 \ s b 


plist.remove( " ' " ) 


With the letter D popped off the front of the list, a call to 
remove dispatehes with the apostrophe: 
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manipulating plist 


What Happened to "plist", Continued 

WeVe been pausing for a moment to consider what actually happened to 
plist as the code in panic . py executed. 

Based on the execution of the code from the last page, we now have a six- 
item list with the characters o, n, t, space, p, and a available to us. Let 5 s 
keep executing our code: 

The Code 


The State of plist 



4 

3 

2 

1 

0 


List 


This is what plist looks like as a resuit of the code on 
the previous page executing: 



The next line of code contains three method 
invocations: two calls to pop and one to extend. The 



All that 5 s left to do (to plist) is to swap the t character 
at location 2 with the space character at index location 3. 
The next line of code contains two method invocations. 
The first uses pop to extract the space character: 

































































































































data 


Lists: What Wc Kwow 

We’re 20 pages in, so let’s take a little break and review what weVe 
learned about lists so far: 



BULLET POINTS 


■ Lists are great for storing a collectiori of 
related objects. If you have a bunch of 
similar things that you’d like to treat as 
one, a list is a great place to put them. 

■ List are similar to arrays in other 
languages. However, unlike arrays in 
other languages (which tend to be fixed 
in size), Python’s lists can grow and 
shrink dynamically as needed. 

■ In code, a list of objects is enclosed in 
square brackets, and the list objects are 
separated from each other by a comma. 


An empty list is represented like this: [ ]. 

The fastest way to check whether an 
object is in a list is to use Python’s in 
operator, which checks for membership. 

Growing a list at runtime is possible 
due to the inclusion of a handful of list 
methods, which include append, 
extend, and insert. 

Shrinking a list at runtime is possible 
due to the inclusion of the remove and 
pop methods. 




Yes. Care is always needed. 

As working with and manipulating lists 
in Python is often very convenient, care 
needs to be taken to ensure the interpreter 
is doing exactly what you want it to. 

A case in point is copying one list to 
another list. Are you copying the list, or 
are you copying the objects in the list? 
Depending on your answer and on what 
you are trying to do, the interpreter will 
behave differently. Flip the page to learn 
what we me an by this. 
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be careful copying 


What Looks like a Copy, Put l$n't 


When to comes to copying an existing list to another one, it’s tempting to use 
the assignment operator: 


»> first = [1, 2, 3, 4, 5] 
»> first 
[1, 2, 3, 4, 5] 

»> second = first 
»> second 
[1, 2, 3, 4, 5] 



Cveate a *ew list (and assia* 
+ive numbeir objedts io it). 


The "-fivst" list's -five numbers 


."Cop y" "the e*isti«5 list to 3 
«ew one, ealled "sedond ■ 


"sedond” list’* 


The sed® 


So far, so good. That looks like it worked, as the five number objects from 
first have been copied to second: 


first 



second 


Or, have they? Let 5 s see what happens when we append a new number to 
second, which seems like a reasonable thing to do, but leads to a problem: 

»> second.append(6) 

»> second 

[1, 2, 3, 4, 5, 6] -- seems OK, but isn't- 


Again, so far, so good—but there’s a bug here. Look what happens when we 
ask the shell to display the contents of first—the new object is appended 
to first too! 


»> first Whoops/ The «ew 

[1, 2, 3, 4, 5, 6] - objett is appe^ed to 

Tirst too. 


/ 



first 



< # second 


This is a problem, in that both first and second are pointing to the same 
data. If you change one list, the other changes, too. This is not good. 
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list data 


How to Copy a Pata Structure 

If using the assignment operator isn’t the way to copy one list to another, 
what is? What’s happening is that a reference to the list is shared among 
first and second. 



1 

' 

n 

■ 

X 

■ 

4- 

c; 

L 




0 

T 

V 

O 



To solve this problem, lists come with a copy method, which does the right 
thing. Take a look at how copy works: 


»> third = second. copy () 

»> third 

[1, 2, 3, 4, 5 , 6] 


first 



second 


third 


in 

dJ 

ijj 

±) 

jj 



With third created (thanks to the copy method), let’s append an object to 
it, then see what happens: 


»> third.append(7) 

The “t^rd” ' ,s ^ »> third 

has ywrnty- ^[1, 2, 3, 4, 5, 6, 7] 

one objefrt. >» seconc j 


[1, 2, 3, 4, 5, 


6] 4 


h 

/Wudh better. The existing 
list is unehah^ed- 


Don’t use the 
assignment 
operator to copy a 
list; use the "copy” 
method instead. 


first 


third 



second 



Thais more like ii—ihe 
*ew objeei is o*!y added io 
ihe third lisi, Koi io ihc 
o-fcheir two lists ("-first" and 
"sedond"). 
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give me brackets 


Square Prackets Are Every where 


O 




Python supports the square bracket 
notation, and then some. 

Everyone who has used square brackets with an 
array in almost any other programming language 
knows that they can access the first value in an array 
called names using names [ 0 ] . The next value 
is in names [ 1 ], the next in names [ 2 ], and so 
on. Python works this way, too, when it comes to 
accessing objects in any list. 

However, Python extends the notation to improve 
upon this standardized behavior by supporting 
negative index values (-1, -2, -3, and so on) as 
well as a notation to select a range of objects from 
a list. 


Lists: Updatiwg What We Already Khow 

Before we dive into a description of how Python extends the square bracket 
notation, let 5 s add to our list of bullet points: 



BULLET POINTS 


■ Take care when copying one list to another. If you want to have another variable reference an existing list, 
use the assignment operator (=). If you want to make a copy of the objects in an existing list and use them 
to initialize a new list, be sure to use the copy method instead. 
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Lists Extewd the Square Pracket Notation 

All our talk of Python’s lists being like arrays in other programming languages wasn’t 
just idle talk. Like other languages, Python starts counting from zero when it comes to 
numbering index locations, and uses the well-known square bracket notation to 
access objects in a list. 

Unlike a lot of other programming languages, Python lets you access the list relative to 
each end: positive index values count from left to right, whereas negative index values 
count from right to left: 


object 


object 


object 


object 


object 


List 


4 

3 

2 

1 

0 


Python's lists undevstand 
positive index values, 
y/With stavt tron> 



0 

1 

1 

l 

4- 

5 

i> 

1 

0 

4 

10 

II 

' 

V 

1 

o 

• 

Y\ 

-- 

) 

1 

t 


' 

F 

. 

3 1 

n 

1 

i 

' 

t 

1 

. 

-11 

-II 

-IO 

-1 

-0 

-1 

-b 


-4- 

-i 

-1 

-1 


Let’s see some examples while working at the shell: 




Create a list ot lettems. 


»> saying = "Don't panic!" 

»> letters = list (saying) 

»> letters 

['D', 'o', 'n', 't', ' 'p', 'a', 'n', 'i', 

»> letters [ 0 ] 


i r» i 


»> letters [3] r *kr 


n i n 


Msmg positive index values tounts 
•fvom le-ft to vight- 


»> letters [6] 


! T 


»> letters [-1] 

' i ' 

»> letters [-3] 
'i' 

»> letters [-6] 

! ! 


T 


...wKeveas negative index values 
tount vigVit to leti- 


J 


• as wcll as negative ihdex 
- values, whieh s-fcar-fc -from 


1 c f f 


* i » 


! ! 1 


It s easy to get at 
the -Pirst a*d last 
objects 'm a*y list- 


As lists grow and shrink while your Python code executes, being able to 
index into the list using a negative index value is often useful. For instance, 
using -1 as the index value is always guaranteed to return the last object 
in the list no matter how big the list is, just as using 0 always returas the first 
object. 

Python’s extensions to the square bracket notation don’t stop with support 
for negative index values. Lists understand start, stop, and step, too. 


»> first 
»> last : 
»> first 
'D' 

»> last 

» i » 


= letters[0] 
: letters[-1] 
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start stop step 


lists Uwdcrstawd Start Stop, awd Step 

We first met start, stop, and step in the previous chapter when discussing the three- 
argument version of the range function: 


0 O 


beereong.py - /Dsers/Paul/Desktop/.NewBook/chO^i* „„ 


word = TT bottles M 

for beer_num in range (99, 0, -1): 
print (beer_num, word, 
print (beer_num, word. 


I 


"of beer 01 A 
"of beer.")' 


ii 


io Vange” 

t^kes threc 
argumcivts; 
OhC eaeh -po' 
s ta\rfc, stop, 
5hd step. 



4 

3 

2 

1 

0 


List 


Recall what start, stop, and step mean when it comes to specifying ranges (and let 5 s 
relate them to lists): 



The START value lets you control WHERE the range begins 

When used with lists, the start value indicates the starting index value. 



The STOP value lets you control WHEN the range ends. 

When used with lists, the stop value indicates the index value to stop at, but not include. 



The STEP value lets you control HOW the range is generated 

When used with lists, the step value refers to the stride to take. 


You can put start stop, and step inside square brackets 

When used with lists, start, stop, and step are specified within the square brackets and 
are separated from one another by the colon (:) character: 


letters [start:stop:step] 

It might seem somewhat counterintuitive, but all three values are optional when used 
together: 



When start is missing, it has a default value of 0. 


The s<\uave 

bv-ac.ket 
notation is 
e%tended 
Work with 
stav-t, stop, 
and step. 


When stop is missing, it takes on the maximum value allowable for the list. 


When step is missing, it has a default value of 1. 
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data 


List Slices in Aetion 


Given the existing list letters from a few pages back, you can specify 
values for start, stop, and step in any number of ways. 

Let 5 s look at some examples: 


»> letters 

['D', 'o', 'n', "' 't', 



AII ihc Ieitev-s 



object 


object 


object 


object 


object 


4 

3 

2 

1 

0 


List 


»> letters [0 :10 : 3] 
['D', 'p', 'i'] 



Evevy -thiv-d le-fc-fcev up io (bui not 
mdludmg) index lodation IO 


»> letters [3:] 

[ II ! II 1 J- 1 I I t I 

r L a r tr r 



n 



Skip the 4iv-s-t thv-ee lettew, 
tben give me evenything e lse. 


»> letters [: 10] 

[ ’ D ’ , 'ob 'n', " ,n f 't', 



n 



AII Ic-tWs uf -fco (bu-t 

loda-tio* IO 


»> letters [: : 2] 

['D', 'n', 't', 'p', 'n', 'c'] <£- Evcvy sc^d Icttcv- 


Using the start, stop, step slice notatiori with lists is very powerful (not to 
mention handy), and you are advised to take some time to understand how 
these examples work. Be sure to follow along at your >>> prompt, and feel 
free to experiment with this notation, too. 


the reictre nQ 

Dumb QuestiQns 


I notice that some of the characters on this page are surrounded by single quotes and others by double quotes. Is there some 
sort of Standard I should follow? 

No, there’s no Standard, as Python lets you use either single or double quotes around strings of any length, including strings that contain 
only a single character (like the ones shown on this page; technically, they are single-character strings, not letters). Most Python programmers 
use single quotes to delimit their strings (but thafs a preference, not a rule). If a string contains a single quote, double quotes can be used to 
avoid the requirement to escape characters with a backslash (\), as most programmers find ifs easier to read " ' " than ’ \ ’ ’. Youll see 
more examples of both quotes being used on the next two pages. 
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start stop list 


Startmg and Stopping with Lists 

Follow along with the examples on this page (and the next) at your >>> prompt and 
make sure you get the same output as we do. 

We start by turning a string into a list of letters: 


Tuvn a 
sfarin^ 

irrto 5 
lis-t, -the* 

display 
•the list- 





»> book = "The Hitchhiker's Guide to the Galaxy" 
»> booklist = list (book) 

»> booklist 
[ 'T' , 


' e' 


V_' o' 



'i' 


» 4 - » 


1 c 1 






» n y 


1 U 1 


'i' 


» A T 


f f ! r* ! f — f ! T f 

r « r ^ r •*- r 


1 e 1 
1 a 1 


Note that the ori<y*al stvm^ dontamed a 
smojle «\uoie character. Pytho», is s™art 
ej.ov.gh to spot this, a».d surrowr>ds the 
sm gle <\wote character wth double gwotes. 


! A T 


! ! 


! ,, ! 


4 

3 

2 

1 

0 


'k', 
't', 

'y ’ ] 


object 


object 


object 


object 


object 


List 


The newly created list (called booklist above) is then used to select a range of 
letters from within the list: 


»> booklist [0:3] Select the tirst three objects 

['T', 'h', 'e'] Oettev-s) tv-om the list- 


»> join (booklist [0 : 3]) 

'The' 


»> 'join (booklist [-6 :]) 
'Galaxy' 


Tur». the selected ra».ge mto a stri».g ^ith 
Vov. leamed hov. to do »>eav- the end <* the 
4a»>it?y” tode) The seCond e*amf>le seledts 
the last sin objects tro» the list- 


Be sure to take time to study this page (and the next) until you’re confident you 
understand how each example works, and be sure to try out each example within 

IDLE. 

With the last example above, note how the interpreter is happy to use any of the 
default values for start, stop, and step. 
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data 


Stepping with Lists 

Here are two more examples, which show off the use of step with lists. 

The first example selects all the letters, starting from the end of the list (that is, it is 
selecting in reverse ), whereas the second selects every other letter in the list. Note how 
the step value Controls this behavior: 



4 

3 

2 

1 

0 


»> backwards = booklist [: :-1] 

»> 1 1 . join (backwards) 

"yxalaG eht ot ediuG s f rekihhctiH ehT" 


»> every_other = booklist[::2] 
»> 1 1 .join(every_other) 
"TeHthie' ud oteGlx" 


List 



Looks like gobbledegook, 
docsh t it? But it is a^tually 
the ov-iginal stirihg \revevsed- 



d tW,s loob like ybberiA! a 

tvo. Jf. ,?• debJW. 


Two final examples confirm that it is possible to start and stop anywhere within the list 
and select objects. When you do this, the returned data is referred to as a slice. Think 
of a slice as a fragment of an existing list. 

Both of these examples select the letters from booklist that spell the word 
' Hitchhiker '. The first selection is joined to show the word ' Hitchhiker ', 
whereas the second displays 'Hitchhiker’ in reverse: 


»> 1 1 . join (booklist [4 :14] ) 
1 Hitchhiker' 


Sli£e out the 
wovd “ttitdhhikev” 


A "slice” is 

a fragment 
of a list. 


»> 'join (booklist [13 : 3 :-1]) 

'rekihhctiH' ^ 

l^Slide owi tbe v/ovd "Hiti-taikev , tat 


do rt m vevevse ov-dev- (i e-, badkwavd)- 


Sliccs are everywhere 


The slice notation doesn’t just work with lists. In fact, you’11 find that you can slice any 
sequence in Python, accessing it with [start: stop: step ]. 


you are here ► 


79 


























panic some more 


Puttiwg Slices to Work ow Usts 

Python’s slice notation is a useful extension to the square bracket notation, 
and it is used in many places throughout the language. You’11 see lots of uses 
of slices as you continue to work your way through this book. 

For now, let’s see Python’s square bracket notation (including the use of slices) 
in action. We are going to take the panic . py program from earlier and 
refactor it to use the square bracket notation and slices to achieve what was 
previously accomplished with list methods. 

Before doing the actual work, here’s a quick reminder of what panic . py 
does. 


object 


object 


object 


object 


object 


List 


4 

3 

2 

1 

0 


Converting "l?on't panic!" to "on tap" 

This code transforms one string into another by manipulating an existing list 
using the list methods. Starting with the string "Don' t panic ! ", this code 
produced "on tap" after the manipulations: 



This is 
U j>ahi£.py 


pisplay -the mftial 

state o£ -the sbrmj-^ 

and list 


Use a £olle£tion ot list methods 
to tv*anstoV*m and manipulatc — 
the list o-f objedts. 


Pisplay the 
vesultm^ state o-f 
the string 3*d list 





phrase = "Don't panic!" 
plist = list (phrase) 
print(phrase) 
print(plist) 
for i in range(4): 

plist.pop() 
plist.pop(0) 
plist.remove) 

plist.extend([plist.pop() , plist.pop()]) 
plist.insert(2, plist.pop(3)) 
new_phrase = ' join (plist) 
print(plist) 
print(new phrase) 


Here’s the output produced by this program when it runs within IDLE: 


Python 3.4.3 Shell 

»> ================================ RESTART =============== 

»> 

Don't panic! 

v , o , n , / t, , p,a,n,i,c,!] 

O / n , , t , a , p ] 

on tap 
»> 


yhe sfarinj "Doni yaWitl” is tva»s-Po*-med in-fco 
"oin taf" tha*b io the list methods. 
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Puttiwg Sliccs to Work ow U$t$, Cowtiwucd 

It 5 s time for the actual work. Here’s the panic . py code again, with the code 
you need to change highlighted: 


These are tV>e 
ot tode 70« need - 


( 


phrase = "Don't panic!" 
plist = list (phrase) 
print(phrase) 
print(plist) 
for i in range(4): 

plist.pop() 
plist.pop(0) 
plist.remove( " ' " ) 

plist.extend([plist.pop(), plist.pop ()]) 
plist.insert (2, plist.pop(3)) 
new_phrase = join (plist) 
print (plist) 
print(new phrase) 



4 

3 

2 

1 

0 


List 


r t^harpen your pencil 


For this exercise, replace the highlighted code above with new code that takes 
advantage of Python’s square bracket notation. Note that you can stili use list 
methods where it makes sense. As before, you’re trying to transform ”Don' t 
panic ! " into "on tap". Add your code in the space provided and call your 
new program panic2 .py: 


phrase = "Don't panic!" 
plist = list (phrase) 
print(phrase) 
print(plist) 


print(plist) 
print(new phrase) 


you are here ► 
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don’t panic again 



en your pencil 
Solutiori 


For this exercise, you were to replace the highlighted code on the previous page 
with new code that takes advantage of PythorTs square bracket notatiori. Note 
that you can stili use list methods where it makes sense. As before, you’re trying 
to transform "Don' t panic ! " into "on tap". You were to call your new 
program panic2. py: 


phrase = "Don't panic!" 
plist = list(phrase) 
print (phrase) 
print (plist) 




We startcd by slidih^ out tbc 
wovd "oh" "plist"... 


»e>M_phvase — new_phrase + * , .jom(CplistC53 ) plistC^Ht plistr?J, plistCOJ) 


print(plist) 
print(new phrase) 


. \ . 

.the* pitked owt eath additioni Jetter that 

M e needed ; spade, "t", "a", a«d p • 


I wonder which of these 
two programs—"panic.py" 
or "panic2.py"—is better? 



That’s a great question. 

Some programmers will look at the code 
inpanic2.py and, when comparing it 
to the code in panic . py, conclude that 
two lines of code is always better than 
seven, especially when the output from 
both programs is the same. Which is a 
fine measurement of “betterness,” but not 
really useful in this case. 

To see what we mean by this, let’s take 
a look at the output produced by both 
programs. 
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Use IDLE to open panic. py and panic2 . py in separate edit Windows. Select the panic. 
py window first, then press F5. Next select the panic2 . py window, then press F5. Compare the 
results from both programs in your shell. 


panic.py - /Users/Paul/Desktop/_NewBook/ch02/panic.py (3.4.3) 



Notice how different these outputs are. 
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which panic? 


T 4- T 


Which Is Petter? It Pepcwds... 

We executed both panic . py and panic2 . py in IDLE to help us 
determine which of these two programs is “better.” 

Take a look at the second-to-last line of output from both programs: 


»> 

Don f t panic! 
['D', 'o', 'n', 

['O', 'n', ' 

on tap 

>» =========== 

>» 

Don't panic! 

['D', 'o', 'n', 

['D', 'o', 'n', 

on tap 

>» 


n i n 



't', 'a', 'p'] 


RESTART 


n i »» 


»» i i» 


't', ' 'p', 'a', 'n', 

't', ' 'p', 'a', 'n'. 


Although both programs conclude by displaying the string "on tap" 
(having first started with the string "Don' t panic!"), panic2.py does 
not change pii st in any way, whereas panic . py does. 

It is worth pausing for a moment to consider this. 

Recall our discussion from earlier in this chapter called “What happened to 
\'plist That discussion detailed the steps that converted this list: 


This is the 
output 
produded bv 

ll i)' 

pamd.py"... 


'c' , 


' I ' 


! '] 


'i' 

'i' 


'c' , 
'c' , 


' ! '] 
' ! '] 



' 

' 

' 

' 

' 


i 


' 

' 

' 

' 

V 

o 

Y\ 

) 

t 


P 

a 

. 

Y\ 

i 

- 

t 

1 

. 

0 

1 

z 

l 

4- 

«5 

t 

i 

0 

1 

10 

II 




The "panid.py" 
pv-oftva» stav-ted 
iwith this list— 


into this much shorter list: 


• 

-, 


1 

' 

■ 

o 

Y\ 


t 

a 

P 









•ahd tuv-hcd it iirfco 
this ohC- 


0 1 z i 4- 5 


All those list manipulations using the pop, remove, extend, and insert 
methods changed the list, which is fine, as that’s primarily what the list 
methods are designed to do: change the list. But what about panic2 .py? 
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Slicing a List Is Nondestructive 


The list methods used by the panic . py program to convert one string into 
another were destructive, in that the original state of the list was altered 
by the code. Slicing a list is nondestructive, as extracting objects from an 
existing list does not alter it; the original data remains intact. 



The 

probram stav-ted 
wrth “this list 


object 


object 


object 


object 


object 


List 


4 

3 

2 

1 

0 


The slices used by panic2 . py are shown here. Note that each extracts data 
from the list, but does not change it. Here are the two lines of code that do all 
the heavy lifting, together with a representation of the data each slice extracts: 


new_j?hrase = ' ' . join (plist [1: 3]) 

new_j?hrase = new_j?hrase + ' join ([plist [5] , plist[4] , plist[7] , plist[6]]) 



The 

Y\OY\ 


diestvwttWe 


slifces 



plist[1:3] 



plist[5] 


plist[4] 


i 


plist[7] 


a 


plist[6] 


P 


The w yanidZ-fy w 
probra** ended uf Vrth 
this list (i-c- r\o dhan^e) 


> 

1 

V 

1 

o 

1 

Y\ 

i 

t 

' 

i 


1 

F 

1 

a 

. 

' 

Y\ 

i 

i 

] 

t 

' 

1 


0 

1 

2. 

l 

4- 

6 

b 

i 

0 

1 

IO 

II 


So...which is better? 

Using list methods to manipulate and transform an existing list does just that: 
it manipulates and transforms the list. The original state of the list is no longer 
available to your program. Depending on what you’re doing, this may (or 
may not) be an issue. Using Python’s square bracket notation generally does 
not alter an existing list, unless you decide to assign a new value to an existing 
index location. Using slices also resuits in no changes to the list: the original 
data remains as it was. 

Which of these two approaches you decide is “better” depends on what you 
are trying to do (and it’s perfectly OK not to like either). There is always 
more than one way to perform a computation, and Python lists are flexible 
enough to support many ways of interacting with the data you store in them. 

We are nearly done with our initial tour of lists. There 5 s just one more topic 
to introduce you to at this stage: list iteration. 


List methods 
change the state 
ol a list, whereas 
using scruare 
hrackets and slices 
(typically) does not. 


you are here ► 
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for loves lists 


Pythow'$ "for" Loop Uwdcrstawds Usts 

Python 5 s for loop knows all about lists and, when provided with any list, knows 
where the start of the list is, how many objects the list contains, and where the 
end of the list is. You never have to teli the for loop any of this, as it works it 
out for itself. 

An example helps to illustrate. Follow along by opening up a new edit window 
in IDLE and typing in the code shown below. Save this new program as 
marvin . py, then press F5 to take it for a spin: 



4 

3 

2 

1 

0 


List 


this 

small probra™ 


■io pv-odude -this ou-fcpn-t. 



O marvin.py - /Users/Paul/Desktop/NewBook/ch02/marvin.py (3.4.3) 


I 


paranoid_android = "Marvin" 
letters = list (paranoidandroid) 
for char in letters: 
print ( '\t r , char) 


Python 3.4.3 (v3.4.3:9b73flc3e 

[GCC 4.2.1 (Apple Inc. build 5___ r ,_ __ _ _ 

Type "Copyright", "credits" or "license()" for more information. 
»> ================================ RESTART =================== 

»> 


Ln: 6 


Coi: 0 



Ea<* tharaeter U the “letters" list is 
printed oy \ its ovm li*e, preeeded by a tab 
dharaeter (that s what the \t does). 


»> 


Ln: 10 Coi: 0 


Uwderstawdmg warviw.py's code 

The first two lines of marvin . py are familiar: assign a string to a variable (called 
par ano id android), then turn the string into a list of character objects 
(assigned to a new variable called letters). 

It’s the next statement—the for loop—that we want you to concentrate on. 

On each iteration, the for loop arranges to take each object in the letters 
list and assign them one at a time to another variable, called char. Within the 
indented loop body char takes on the current value of the object being processed 
by the for loop. Note that the for loop knows when to start iterating, when to 
stop iterating, as well as how many objects are in the letters list. You don’t need 
to worry about any of this: that’s the interpreter’s job. 


cadh iteratio*, 
s variable 
rers to “the 
rre*t objedt 


This is the list to 
iterate over. 



for char in letters: 

print( 1 \t 1 , char) 

k.-- - ; 

f 

This blotk o£ tode 
e%e£utes o* eafch iteratio*- 
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Pythow'$ "for" Loop Uwdcrstawds Slices 

If you use the square bracket notation to select a slice from a list, the for loop “does 
the right thing” and only ite rates over the sliced objects. An update to our most recent 
program shows this in action. Save a new version of marvin . py as marvin2 . py, 
then change the code to look like that shown below. 

Of interest is our use of Python’s multiplication operator (*), which is used to 
control how many tab characters are printed before each object in the second and 
third for loop. We use * here to “multiply” how many times we want tab to appear: 


c . \ 

object 



object 2 


object 1 
object 0 

List 


# <§> # marvin2.py - /Users/Paul/Desktop/_NewBook/ch02/marvin2 J py (3.4,3) 



you are here ► 
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for loop slices 


Marvin's Slices in Petail 

Let’s take a look at each of the slices in the last program in detail, as this 
technique appears a lot in Python programs. Below, each line of slice code 
is presented once more, together with a graphical representation of what 5 s 
going on. 

Before looking at the three slices, note that the program begins by assigning a 
string to a variable (called paranoid_android) and converting it to a list 
(called letters): 

paranoid_android = "Marvin, the Paranoid Android" 
letters = list(paranoid_android) 


object 


object 


object 


object 


object 


4 

3 

2 

1 

0 


List 


letters 


Redall -tha-t yo* £ 3 * a£dess a*y 

Slot in a list <*si«5 a ?os4Wc or 
«elative inde* valwe- Vk rt onty 

sViovjm^ some ot tVie ne^at-we - 
inde* valwes iieve- 


O I 


/VI 


jJ 


' 

o 

■-! 

i 

d 

'-1 

i 

Y\ 

d 

' 

| 

O 

i 

10 

n 

10 

11 

ii 

11 

24 


u 

zi 

-II 

-10 

-7 

-e 

-i 

-T 

-5 

-4 

-l 

-1 


-I 


i 

4 

5 

i 

7 

e 

i 

10 

a 

li 

ii 

14 

' ' 

1 ? 


17 

V 

1 

i 

Y\ 

1 J 

■ 

i 

■ 

h 

e . 

• 

p 

a 

V 

a 

V 

Y\ 


We’11 look at each of the slices from the marvin2 . py program and see what 
they produce. When the interpreter sees the slice specification, it extracts the 
sliced objects from letters and returns a copy of the objects to the for 
loop. The original letters list is unaffected by these slices. 

The first slice extracts from the start of the list and ends (but doesnl include) 
the object in slot 6: 

for char in letters [: 6]: letters[:6] # 

print('\t', char) 


/H 


□ 



■ 


V 

i 

Y\ 





The second slice extracts from the end of the letters list, starting at slot -7 
and going to the end of letters: 


for char in letters[-7:]: 
print('\t'*2, char) 


letters[-7:] # > 


And finally, the third slice extracts from the middle of the list, starting at slot 
12 and including everything up to but not including slot 20: 

for char in letters[12:20]: 

print (' \t' *3, char) letters[12:20] # > 
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data 


Usts: Updatiwg What We Kwow 

Now that youVe seen how lists and for loops internet, let’s quickly review 
what youVe learned over the last few pages: 



BULLET POINTS 


■ Lists understand the square bracket notation, 
which can be used to select individual objects 
from any list. 

■ Like a lot of other programming languages, 
Python starts counting from zero, so the first 
object in any list is at index location 0, the 
second at 1, and so on. 


Unlike a lot of other programming languages, 
Python lets you index into a list from either end. 
Using -1 selects the last item in the list, -2 the 
second last, and so on. 

Lists also provide slices (or fragments) of a list 
by supporting the specification of start, stop, 
and step as part of the square bracket notation. 


0 


Oo 




are used a lot, but... 

They are not a data structure panacea. Lists can be used in lots 
of places; if you have a collection of similar objects that you 
need to store in a data structure, lists are the perfect choice. 

However—and perhaps somewhat counterintuitively—if the 
data you’re working with exhibits some structure , lists can be a 
bad choice. We’ll start exploring this problem (and what you 
can do about it) on the next page. 


thereictre no 

Diimb Oiiestions 



Surely there’s a lot more to lists than this? 


Yes, there is. Think of the material in this chapter as a quick 
introduction to Python’s built-in data structures, together with what 
they can do for you. We are by no means done with lists, and will be 
returning to them throughout the remainder of this book. 



But what about sorting lists? Isrft that important? 


Yes, it is, but lefs not worry about stuff like that until we actually 
need to. For now, if you have a good grasp of the basies, thafs all 
you need at this stage. And don’t worry: we’ll get to sorting soon. 


you are here ► 
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not a panacea 


What'$ Wrong with lists? 

When Python programmers find themselves in a situation where they need to 
sto re a collection of similar objects, using a list is often the natural choice. After all, 
weVe used nothing but lists in this chapter so far. 

Recall how lists are great at storing a collection of related letters, such as with the 
vowels list: 


vowels = [ f a f , f e f , f i f , f o f , f u f ] 

And if the data is a collection of numbers, lists are a great choice, too: 

nums = [1, 2, 3, 4, 5] 


In fact, lists are a great choice when you have a collection of related anythings. 


But imagine you need to store data about a person, and the sample data you Ve 
been given looks something like this: 


$e*der : Ma\e 

Re sea ' rtV ' ev ’ 

fo** PUet 





daia -Pov 
you io play wj-fch 



On the face of things, this data does indeed conform to a structure, in that there’s 
tags on the left and associated data values on the right. So, why not put this data in a 
list? After all, this data is related to the person, right? 

To see why we shouldn’t, let’s look at two ways to store this data using lists (starting 
on the next page). We are going to be totally upfront here: both of our attempts 
exhibit problems that make using lists less than ideal for data like this. But, as the 
journey is often half the fun of getting there, we’re going to try lists anyway. 

Our first attempt concentrates on the data values on the right of the napkin, 
whereas our second attempt uses the tags on the left as well as the associated data 
values. Have a think about how you’d handle this type of structured data using 
lists, then flip to the next page to see how our two attempts fared. 
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data 


Whew Not to Use Usts 

We have our sample data (on the back of a napkin) and weVe decided to store 
the data in a list (as that’s all we know at this point in our Python travels). 

Our first attempt takes the data values and puts them in a list: 

»> personi = [ ! Ford Prefect' , 'Male', 

1 Researcher 1 , 1 Betelgeuse Seven 1 ] 

»> personi 

[' Ford Prefect', 'Male ', 'Researcher', 

1 Betelgeuse Seven 1 ] 

This results in a list of string objects, which works. As shown above, the shell 
confirms that the data values are now in a list called personi. 

But we have a problem, in that we have to remember that the first index 
location (at index value 0) is the person 5 s name, the next is the person’s gender 
(at index value 1), and so on. For a small number of data items, this is not 
a big deal, but imagine if this data expanded to include many more data 
values (perhaps to support a profile page on that Facebook-killer you 5 re been 
meaning to build). With data like this, using index values to refer to the data 
in the personi list is brittle, and best avoided. 

Our second attempt adds the tags into the list, so that each data value is 
preceded by its associated tag. Meet the person2 list: 


Does "person[l]" 
refer to gender or 
occupation? I can 
never remember! 


O 

o 


Ndmc : Fovd Pv-etedt 
gender: Male 
Oddupation; Reseav-dtev 
ttowe Planet Betelgeuse Seven 


»> person2 = ['Name', f Ford Prefect' , ' Gender' , 

'Male ', 'Occupation', 'Researcher', 1 Home Planet 1 , 

1 Betelgeuse Seven 1 ] 

»> person2 

[ 1 Name 1 , 1 Ford Prefect 1 f 1 Gender 1 , 1 Male 1 , 

1 Occupation 1 , 1 Researcher 1 , ! Home Planet 1 , 

1 Betelgeuse Seven 1 ] 

This clearly works, but now we no longer have one problem; we have two. 

Not only do we stili have to remember what 5 s at each index location, but we 
now have to remember that index values 0, 2, 4, 6, and so on are tags, while 
index values 1, 3, 5, 7, and so on are data values. 

Surely there has to be a better way to handle data with a structure like this? 

There is, and it involves foregoing the use of lists for structured data like this. 

We need to use something else, and in Python, that something else is called a 
dictionary, which we get to in the next chapter. 


II the data you 
want to store has 
an ictentiliahte 
structure, consicter 
using something 
other than a list. 


you are here > 
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the code 


Chapter 2's Code, 1 of 1 


vowels = ['a', 'e', 'i', ' o', ' u'] 

word = "Milliways" 
for letter in word: 

if letter in vowels: 
print(letter) 



• ne -riv-st version o\ -the 
vowels progra», that displays 
all^ -the vowels fouhd i h -the 
vord “Milliways" UcU,** a„y 
duplicates). 3 1 


The 'WUZ-py” pv-o^vam 

added dode that <*sed a list 
-to avoid dvplitates. This 
pvo^v-a» displays -the list °+ 
wr ,i - ye vowels -foiAhd i» the 
word "Milliways". 


—^ 


vowels = ['a', 'e', 'i', 'o', 'u'] 

word = "Milliways" 

found = [] 

for letter in word: 

if letter in vowels: 

if letter not in found: 
found.append(letter) 
for vowel in found: 
print(vowel) 


vowels = ['a', 'e', 'i', 'o', 'u'] 

word = input("Provide a word to search for vowels: ") 

found = [] 

for letter in word: 

if letter in vowels: 

if letter not in found: 
found.append(letter) 
for vowel in found: 
print(vowel) 


The thiird PinaU versior, 

o\ the vowels progvar* •for 
this chapter, "vowelslpy", 
displays the unigue vowels 

touhd 'm a word ehtered by 
ouv- usev-. 


It’s the best advide W> the uhiverse^ Poh t 
parit!" This probam, talled pamt-py , 
takes a strihg dohtamirg this adv.te a«d, 
,sih 5 a Whdh of list «ethods, Wor,s 
the strih^ ihto arother sbrfoft that 
desdribes how the rtead fiwt cd.tors 
pve-fer their beer. "<m tap • 




phrase = "Don't panic!" 
plist = list (phrase) 
print(phrase) 
print(plist) 

for i in range(4): 

plist.pop() 
plist.pop(0) 
plist.remove( " ' " ) 

plist.extend([plist.pop (), plist.pop()]) 
plist.insert(2, plist.pop (3)) 

new_phrase = join (plist) 
print(plist) 
print(new phrase) 
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data 


Chapter 2's Code, 1 of 1 


phrase = "Don't panic!" 
plist = list (phrase) 
print(phrase) 
print(plist) 

new_phrase = ' ' .join (plist [ 1:3]) 

new_phrase = new_phrase + ' ' .join ( [plist [5], plist[4], plist [7], plist[6]]) 

print(plist) 
print(new phrase) 


When \i t™* -to uS '^ 

„,ethods isn't the >Y 5 a " e TV>e . 

> a ritzrf atWie t d J t 7T e e " d 

usmO) Python’* brai-ket notaW- 



paranoid_android = "Marvin" 
letters = list(paranoid_android) 
for char in letters: 
print('\t', char) 



- r, T/ i wcmohStvatCd how wcll 

lists pUy with Python’s "4/' Ioop. 

Oust doh t teli /l/|awin...i-f he hea« 

that His fvogira* is the shoi-test in 

,s (■ a ptev, it II make him even n>ove 
pairanoid than he alv-eady is). 


The "*>airvinZ.?y” ?' r °r a "' 
shov/ed oW Python’s sc y* av T 
bvafcket notation by thvee 
slites 4 e*tvatt and display 
4agments tom a list o+ 
lettevs- 


paranoid_android = "Marvin, the Paranoid Android" 
letters = list(paranoid_android) 
for char in letters[:6]: 

print( 1 \t 1 , char) 
print() 

for char in letters[-7:]: 

print( 1 \t 1 *2, char) 
print() 

for char in letters[12:20]: 
print(’\t 1 *3, char) 


you are here ► 
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3 sttuctured data 


^ Working with Structured Data + 



Python’s list data structure is great, but it isn’t a data panacea. 

When you have truly structured data (and using a list to store it may not be the best 
choice), Python comes to your rescue with its built-in dictionary. Out of the box, the 
dictionary lets you store and manipulate any collection of key/value pairs. We look long 
and hard at Python’s dictionary in this chapter, and—along the way—meet set and tuple, 
too. Together with the list (which we met in the previous chapter), the dictionary, set, and 
tuple data structures provide a set of built-in data tools that help to make Python and data 
a powerful combination. 


this is a new chapter 
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key: value 


A Pictiowary Stores Key/Value Pairs 

Unlike a list, which is a collection of related objects, the dictionary is used 
to hold a collection of key/value pairs, where each unique key has a value 
associated with it. The dictionary is often referred to as an associative array by 
computer scientists, and other programming languages often use other names 
for dictionary (such as map, hash, and table). 

The key part of a Python dictionary is typically a string, whereas the 
associated value part can be any Python object. 


key#4 

object 

key#l 

object 

key#3 

object 

key#2 

object 


Dictionary 


Data that conforms to the dictionary model is easy to spot: there are two 
columns, with potentially multiple rows of data. With this in mind, take 
another look at our “data napkin” from the end of the last chapter: 


Here' s one 
toluwn o£ 
data*** 




-and here's the S e£ond 
£olur*h o-f data. 


5 Ue: fVd 

fa&cr-- Male 

OttvY&w Reseavt^ev 



In C++ and Java, a 
dictionary is known 


w 

as m 


ap, whereas 
Perl and Ruhy use 
the name "hask” 


There are 

multae °* 

•bwo-^oluwned 

data oy\ this 

naykin. 


It looks like the data on this napkin is a perfect fit for Python’s dictionary. 


Let’s return to the >>> shell to see how to create a dictionary using our 
napkin data. It’s tempting to try to enter the dictionary as a single line of 
code, but we’re not going to do this. As we want our dictionary code to be 
easy to read, we’re purposely entering each row of data (i.e., each key/value 
pair) on its own line instead. Take a look: 


The na**c 
the ditWary. 
(Retall that 
met V rS0 ’ r 'f , ) , 

and '\emon2-’ at 
the end ot the 
last tha?te*-) 



The assodiated data value 


The key 

l 

»> person3 = { 'Name' : 'Ford Prefect' , 

'Gender': 'Male', 

'Occupation': 'Researcher', 

'Home Planet': 'Betelgeuse Seven' } 


r 

Key 



\/alue 
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structured data 


Make Pictionaries Easy to Read 

It s tempting to take the four lines of code from the bottom of the last page and 
type them into the shell like this: 


»> person3 = { 'Name': 'Ford Prefect', 'Gender': 

'Male', 'Occupation': 'Researcher', 'Home Planet': 

'Betelgeuse Seven' } 


key#4 

object 

key#l 

object 

key#3 

object 

key#2 

object 


Dictionary 


Although the interpreter doesn’t care which approach you use, entering a 
dictionary as one long line of code is hard to read, and should be avoided 
whenever possible. 

If you litter your code with dictionaries that are hard to read, other 
programmers (which includes you in six months’ time) will get upset...so take the 
time to align your dictionary code so that it is easy to read. 

Here 5 s a visual representation of how the dictionary appears in Python’s 
memory after either of these dictionary-assigning statements executes: 



person3 # 


The W ycvs<w3 variati e 

ve-Ceventes *thc cntiv-c 
dietionav-y* v/hieh is 
r*ade up o£ a fcolle^tion 
Jc key/value paivs. 



This is a more complicated structure than the array-like list. If the idea behind 
Python’s dictionary is new to you, it’s often useful to think of it as a lookup 
table. The key on the left is used to look up the value on the right (just like you 
look up a word in a paper dictionary). 

Let’s spend some time getting to know Python’s dictionary in more detail. We’ll 
begin with a detailed explanation of how to spot a Python dictionary in your 
code, before talking about some of this data structure^ unique characteristics 
and uses. 


you are here ► 
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it’s a dictionary 


How to Spot a Pictiowary in Code 

Take a closer look at how we defined the person3 dictionary at the >>> 
shell. For starters, the entire dictionary is enclosed in curly braces. Each key is 
enclosed in quotes, as they are strings, as is each value, which are also strings 
in this example. (Keys and values don’t have to be strings, however.) Each key 
is separated from its associated value by a colon character (:), and each key/ 
value pair (a.k.a. “row”) is separated from the next by a comma: 


key#4 

object 

key#l 

object 

key#3 

object 

key#2 

object 


Dictionary 


J\y\ 

V>\r3dc 
eath di 




I* this did-tiohav-y, the values ave all 

stving objeets, so -they ave enclosed 

m Quotes. 


Name': 'Ford Prefect', 

Gender': 'Male', 

Occupatiori' : ' Researcher' , 

Home Planet': 'Betelgeuse Seven' 

/|\ 



/\ tolon assotiates eath 
key v/rth it* value- 


Each key/value paiv is 
sepavated tvom the »ext 
by a Comma. 



di^tionav-y 


As stated earlier, the data on this napkin maps nicely to a Python dictionary 
In fact, any data that exhibits a similar structure—multiple two-columned 
rows—is as perfect a fit as you’re likely to find. Which is great, but it does 
come at a price. Let 5 s return to the >>> prompt to learn what this price is: 



/\sk the shell 
to disylay the 
Covtents ot the 
dictionavy ••• 


»> person3 

{'Gender': 'Male', 'Name': 

Planet': 'Betelgeuse Seven 


What happewed to the iwsertion order? 


'Ford Prefect', 'Home 

, 'Occupation': 'Researcher'} 

1^- and theve it is. /\H the 

key/value paivs ave shovm. 


Take a long hard look at the dictionary displayed by the interpreter. Did you 
notice that the ordering is different from what was used on input? When you 
created the dictionary, you inserted the rows in name, gender, occupation, 
and horne planet order, but the shell is displaying them in gender, name, 
horne planet, and occupation order. The ordering has changed. 


What’s going on here? Why did the ordering change? 
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structured data 


Insertion Order 1$ NOT Maintained 

Unlike lists, which keep your objects arranged in the order in which you 
inserted them, Python’s dictionary does not. This means you cannot assume 
that the rows in any dictionary are in any particular order; for all intents and 
purposes, they are unordered. 

Take another look at the person3 dictionary and compare the ordering on 
input to that shown by the interpreter at the >>> prompt: 


key#4 

object 

key#l 

object 

key#3 

object 

key#2 

object 


Dictionary 


»> person3 


»> person3 
{'Gender': 

'Betelgeuse 


= { 'Name': 'Ford Prefect', 

'Gender': 'Male', 

'Occupation': 'Researcher', 

'Home Planet': 'Betelgeuse Seven 

'Male', 'Name': 'Ford Prefect', 'Home 

Seven', 'Occupation': 'Researcher'} 



You insevi youv data 
inio a diciionavy in 
one ovdev... 

...tat the intevyvetev 
uses another 
1 } ordering 

Planet f : 



If you’re scratching your head and wondering why you’d want to trust 
your precious data to such an unordered data structure, don’t worry, as 
the ordering rarely makes a difference. When you select data stored in a 
dictionary, it has nothing to do with the dictionary 5 s order, and everything to 
do with the key you used. Remember: a key is used to look up a value. 


Pictiowaries understand square brackets 

Like lists, dictionaries understand the square bracket notation. However, 
unlike lists, which use numeric index values to access data, dictionaries use 
keys to access their associated data values. Let’s see this in action at the 
interpreter’s >>> prompt: 


Provide key 
beWen the s^uave 
taatkeis 


»> person3 [' Home Planet' ] 
'Betelgeuse Seven' 

-V 

»> person3 [' Name' ] 

'Ford Prefect' 


Use keys to 
access data in 
a dictionary. 



The daia value 
Associated with the 

key is shoy/h. 


When you consider you can access your data in this way, it becomes apparent 
that it does not matter in what order the interpreter Stores your data. 


you are here ► 


99 


























dictionaries love brackets 


Value Lookup with Square Prackets 

Using square brackets with dictionaries works the same as with lists. However, 
instead of accessing your data in a specified slot using an index value, with Pythoffs 
dictionary you access your data via the key associated with it. 

As we saw at the bottom of the last page, when you place a key inside a dictionary 5 s 
square brackets, the interpreter returns the value associated with the key Let 5 s 
consider those examples again to help cernent this idea in your brain: 


key#4 

object 

key#l 

object 

key#3 

object 

key#2 

object 


Dictionary 


»> person3 [ 1 Home Planet 1 ] 
1 Betelgeuse Seven 1 






Ma\t 


MdrnC 



Ford Pvc-fcdt 


Home Planek 



Betelgeuse Seven 



»> person3 [ 1 Name ' ] 
' Ford Prefect' 



Pictiowary lookup is fast! 

This ability to extract any value from a dictionary using its associated key is what 
makes Pythoffs dictionary so useful, as there are lots of occasions when doing so is 
needed—for instance, looking up user details in a profile, which is essentially what 
we’re doing here with the person3 dictionary 

It does not matter in what order the dictionary is stored. All that matters is that the 
interpreter can access the value associated with a key quickly (no matter how big 
your dictionary gets). The good news is that the interpreter does just that, thanks to 
the employment of a highly optimized hashing algorithm. As with a lot of Pythoffs 
internals, you can safely leave the interpreter to handle all the details here, while 
you get on with taking advantage of what Pythoffs dictionary has to offer. 



Pythoffs dictionary 
is implemented as a 
resizeable hash table, 
which has been heavily 
optimized for lots of 
special cases. As a resuit, 
dictionaries perform 
lookups very quickly. 
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structured data 


Workiwg with Pictiowaries at Runtiwe 


Knowing how the square bracket notation works with dictionaries is Central to 
understanding how dictionaries grow at runtime. If you have an existing dictionary 
you can add a new key/value pair to it by assigning an object to a new key, which you 
provide within square brackets. 

For instance, here we display the current state of the person3 dictionary then add 
a new key/value pair that associates 33 with a key called Age. We then display the 
person3 dictionary again to confirm the new row of data is successfully added: 


Be-Cov-e -bKc new 
vow is added 


»> person3 



Gender 


Male 


key#4 

object 

key#l 

object 

key#3 

object 

key#2 

object 


Dictionary 




ove 



»> person3 [' Age ' ] =33 


Assign an objei-t (in -this dase, a 

T, , a he * ke Y a v~o\w 

° ^ to 'the did-tionav-y. 



»> person3 
{'Name': 'Ford Prefect', 'Gender': 'Male', 

1 Age ' : 33 , 1 Home Planet': 'Betelgeuse Seven 1 ,^ ^ 

1 Occupatiori 1 : 1 Researcher 1 } A-P-fcer -the ne 

row is added 


vow ©£ data : 

is assodia-ted 

witt tt Ay • 





A-P-fe, 


e\r 


you are here ► 
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remembering vowels3.py 


Recap: Pisplaying Found Vowels (lists) 

As shown on the last page, growing a dictionary in this way can be used 
in many different situations. One very common application is to perform 
a frequency count : processing some data and maintaining a count of what 
you find. Before demonstrating how to perform a frequency count using a 
dictionary, let’s return to our vowel counting example from the last chapter. 

Recall that vowels3 . py determines a unique list of vowels found in a word. 
Imagine you Ve now been asked to extend this program to produce output 
that details how many times each vowel appears in the word. 

Here 5 s the code from Chapter 2, which, given a word, displays a unique list of 
found vowels: 


This is "vowelslf/”> -? 

vihidh veyov-b o» 

■the uni«\i*e vowels 
tound in 3 wovd- 


# O • vowels3.py -/Users/Paul/Desktop/_NewBook/ch02/vowels3.py (3.4.3) 


vowels = [ 'a' , 'e', 'i', 'o', 'u'] 

word = input( 'Provide a word to search for vowels: ) 

found = [] 

for letter in word: 

if letter in vowels: 

if letter not in found: 
found.append(letter) 
for vowel in found: 
print (vowel) 


Ln: 11 Coi: 0 



Recall that we ran this code through IDLE a number of times: 


# # # Python 3.4.3 Shell 

»> ================================ RESTART ================================ 

»> 

Provide a word to search for vowels: Milliways 

• 

x 

a 

»> ================================ RESTART ================================ 

»> 

Provide a word to search for vowels: Hitch-hiker 
• 

x 

e 

»> ================================ RESTART ================================ 

»> 

Provide a word to search for vowels: Galaxy 
a 

»> ================================ RESTART ================================ 

»> 

Provide a word to search for vowels: Sky 
»> 

Ln: 21 Coi: 4 
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structured data 


How Can a Pictionary Help Here? 



t 



s* 


We aren’t. 

The vowels3 . py program does what it is 
supposed to do, and using a list for this version 
of the progranTs functionality makes perfect 
sense. 

However, imagine if you need to not only list 
the vowels in any word, but also report their 
frequency. What if you need to know how many 
times each vowel appears in a word? 

If you think about it, this is a little harder to do 
with lists alone. But throw a dictionary into the 
mix, and things change. 

Let’s explore using a dictionary with the vowels 
program over the next few pages to satisfy this 
new requirement. 


tliere itire no 

- Diimb Questions - 

Is it just me, or is the word “dictionary” a strange name for something that’s basically a table? 

No, ifs not just you. The word “dictionary” is what the Python documentation uses. In fact, most Python programmers use the shorter 
“dict” as opposed to the full word. In its most basic form, a dictionary is a table that has exactly two columns and any number of rows. 


you are here ► 
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what’s the frequency, kenneth? 


Selectiwg a Frequency Count Pata Structure 

We want to adjust the vowels3 . py program to maintain a count of how often each 
vowel is present in a word; that is, what is each vowefs frequency? Let 5 s sketch out 
what we expect to see as output from this program: 


-the »<*A ***’* 

J^uenty 6o«a «e b> se< : 



than using a list to store the found vowels (as is the case in vowels3 . py), let’s use 
a dictionary instead. We can continue to call the collection found, but we need to 
initialize it to an empty dictionary as opposed to an empty list. 


key#4 

object 

key#l 

object 

key#3 

object 

key#2 

object 


Dictionary 


As always, let’s experiment and work out what we need to do at the >>> prompt, 
before committing any changes to the vowels3 . py code. To create an empty 
dictionary, assign { } to a variable: 


»> found = {} 
»> found 

u 


<: -Cuirly brades o« -feheiv om mea>t -the 

didtionavy stav-ts out e»»f>ty. 


Let’s record the fact that we haven’t found any vowels yet by creating a row for each 
vowel and initializing its associated value to 0. Each vowel is used as a key: 


»> found [ 1 a 1 ] = 0 
»> found [ 1 e 1 ] = 0 
»> found [ 1 i 1 ] = 0 
»> found [ 1 o 1 ] = 0 
»> found [ 1 u 1 ] = 0 
»> found 





l/Vc ; ve initialized all the 
vowel £ounts to O. /Vote 
how insevtion order is 
not rwaintained (but 
that doesn t m attev- 
hev*e). 


All we need to do now is find a vowel in a given word, then update these frequency 
counts as required. 
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structured data 


Updating a Frequency Couwter 

Before getting to the code that updates the frequency counts, consider how 
the interpreter sees the f ound dictionary in memory after the dictionary 
initialization code executes: 



key#4 

object 

key#l 

object 

key#3 

object 

key#2 

object 



AII the values 
av-e initially 

set to O . 


Dictionary 


With the frequency counts initialized to 0, it 5 s not difficult to increment 
any particular value, as needed. For instance, here’s how to increment e’s 
frequency count: 


»> found 
{ ! o ! : 0, ! u ! : 0, 

»> found [ 1 e 1 ] = 

»> found 

{, 0 .• q ! i ! * 0 ! 3 ! * 0 ! u ! * 0 


'a': 0, 'i': 0, 

found['e'] +1 


'e': 0} 


'e': 1} 


found 


Ever yfching 
is O. 



. ° ^ 

O 


0 


0 


0 


1 


|h£v-er*eht cs 
£ount* 



The ditWary has bee* 
uydated- The value 
assodiated *»th V has 
been mdve^ented- 



Code like that highlighted above certainly works, but having to repeat 
found [ ' e f ] on either side of the assignment operator gets very old, very 
quickly So, let’s look at a shorteut for this operation (on the next page). 


you are here ► 
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plus equals 


Updatiwg a Frequency Counter, v2.0 

Having to put f ound [ ' e ' ] on either side of the assignment operator 
(=) quickly becomes tiresome, so Python supports the familiar += 
operator, which does the same thing, but in a more succinet way: 


»> f ound [ 1 e 1 ] += 1 
»> found 

{io ! • 0 ! i ! * 0 ! a 1 • 0 


u 


0, 


/h£v-er*eiyfc es 
^ouht (oy\U more). 

1 e 1 : 2} ^-- 


key#4 

object 

key#l 

object 

key#3 

object 

key#2 

object 


Dictionary 


TKc dtetiohary 
is updated 

At this point, weVe incremented the value associated with the e key twice, agaih. 

so here’s how the dictionary looks to the interpreter now: 


found 



Thanks -to the +— operator, 
the value associated with the 
e key has beeh ihCvemehted 


ohde more- 


ttereicire no 

Dumb Questipns 



Does Python have ++? 


No...which is a bummer. If you’re a fan of the ++ increment 
operator in other programming languages, youll just have to get used 
to using += instead. Same goes for the -- decrement operator: 
Python doesn’t have it. You need to use -= instead. 



Is there a handy list of operators? 


Yes. Head over to https://docs.python.Org/3/reference/lexical_ 
analysis.html#operators for a list, and then see https://docs.python. 
org/3/library/stdtypes.html for a detailed explanation of their usage in 
relation to Python’s built-in types. 
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structured data 


Iterating Over a Pictionary 

At this point, weVe shown you how to initialize a dictionary with zeroed 
data, as well as update a dictionary by incrementing a value associated 
with a key. We’re nearly ready to update the vowels3 . py program to 
perform a frequency count based on vowels found in a word. However, 
before doing so, let 5 s determine what happens when we iterate over a 
dictionary, as once we have the dictionary populated with data, wedl 
need a way to display our frequency counts on screen. 

You’d be forgiven for thinking that all we need to do here is use the 
dictionary with a for loop, but doing so produces unexpected results: 


key#4 

object 

key#l 

object 

key#3 

object 

key#2 

object 


Dictionary 


We itevate ovev tbe 
ditWavy 'm tbe wual 
way, a "tov J<*?• 
Heve, weVe usin$ kv as 
skov-tkand U "key/valwe 
r, 3 iy M (but fcould ve used 
any vaviable na me)' 


»> for kv in found: 

print(kv) 


o 

■ 

1 

a 

u 

e 




The itevation wo\rked> but 
•this isn t what we weve 
e^pettin^. lA/heve have the 
-fve^uenty eounts $one? 
This output is only showin^ 
the keys... 


O 



Somethings really not right 
with this output. The keys are 
being displayed, but not their 
associated values. What gives? 


Flip the page to learn what 
happened to the values. 


you are here ► 
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k and found[k] 


Iteratiwg Over Keys and Values 

When you iterated over a dictionary with your for loop, the interpreter only 
processed the dictionary 5 s keys. 

To access the associated data values, you need to put each key within square 
brackets and use it together with the dictionary name to gain access to the 
values associated with the key 

The version of the loop shown below does just that, providing not just the 
keys, but also their associated data values. WeVe changed the suite to access 
each value based on each key provided to the for loop. 


key#4 

object 

key#l 

object 

key#3 

object 

key#2 

object 


Dictionary 


As the for loop iterates over each key/value pair in the dictionary, the 
current row’s key is assigned to k, then f ound [ k] is used to access its 
associated value. WeVe also produced more human-friendly output by 
passing two strings to the call to the print function: 


We re usmg k io repnesen-t 
the key, and "-foundfkJ" -fe, 
addess the value- 


»> for k in found: 

print(k, 'was found' 



O 

was 

found 

0 

time(s) 

■ 

1 

was 

found 

0 

time(s) 

a 

was 

found 

0 

time(s) 

u 

was 

found 

0 

time(s) 

e 

was 

found 

2 

time(s) 



This is more like it- The keys and the 
values are being prodessed by the loop and 
disylayed on stv-een- 


If you are following along at your >>> prompt and your output is ordered 
differently from ours, dofft worry: the interpreter uses a random internal 
ordering as yoffre using a dictionary here, and there are no guarantees 
regarding ordering when one is used. Your ordering will likely differ from 
ours, but dofft be alarmed. Our primary concern is that the data is safely 
stored in the dictionary, which it is. 

The above loop obviously works. However, there are two points that we’d like 
to make. 

Firstly: it would be nice if the output was ordered a, e, i, o, u, as opposed to 
randomly, wouldfft it? 

Secondly: even though this loop clearly works, coding a dictionary iteration in 
this way is not the preferred approach—most Python programmers code this 
differently. 

Leffs explore these two points in a bit more detail (affer a quick review). 
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structured data 


Pictiowaries: What We Already Know 

Here’s what we know about Python’s dictionary data structure so far: 



BULLET POINTS 


■ Think of a dictionary as a collection of rows, with each 
row containing exactly two columns. The first column 
Stores a key, while the second contains a value. 


Insertion order is not maintained by a dictionary. The 
order in which rows are inserted has nothing to do with 
how they are stored. 


Each row is known as a key/value pair, and a dictionary 
can grow to contain any number of key/value pairs. Like 
lists, dictionaries grow and shrink on demand. 


Accessing data in a dictionary uses the square bracket 
notation. Put a key inside square brackets to access its 
associated value. 


A dictionary is easy to spot: ifs enclosed in curly braces 
with each key/value pair separated from the next by a 
comma, and each key separated from its value by a 
colon. 


Python’s for loop can be used to iterate over a 
dictionary. On each iteration, the key is assigned to the 
loop variable, which is used to access the data value. 


Specifying the ordering of a dictionary on output 

We want to be able to produce output from the for loop in a, e, i, o, u 
order as opposed to randomly. Python makes this trivial thanks to the 
inclusion of the sorted built-in function. Simply pass the f ound dictionary 
to the sorted function as part of the for loop to arrange the output 
alphabetically: 


»> for k in sorted (found) : 

print(k, 'was found' Afound[k] , f time(s) . ! ) 


a 

was 

found 

0 

time(s) 

e 

was 

found 

2 

time (s) 

■ 

i 

was 

found 

0 

time (s) 

o 

was 

found 

0 

time(s) 

u 

was 

found 

0 

time(s) 


tht ^ w 

. F , k d u '^ ihe i-ook: ihe ouW 

' s soried i» a, e, i, O, u" order. ? 


That’s point one of two dealt with. Next up is learning about the approach 
that most Python programmers prefer over the above code (although the 
approach shown on this page is often used, so you stili need to know about it). 


you are here ► 
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the items idiom 


Iterating Over a Pictionary with "items" 

WeVe seen that it’s possible to iterate over the rows of data in a dictionary using this 
code: 

»> for k in sorted (found) : 

print(k, 'was found 1 , found[k], f time(s). f ) 


a 

was 

found 

e 

was 

found 

■ 

i 

was 

found 

o 

was 

found 

u 

was 

found 


0 time (s) . 
2 time (s) . 
0 time (s) . 
0 time (s) . 
0 time (s) . 


Like lists, dictionaries have a bunch of built-in methods, and one of these is the 
items method, which returns a list of the key/value pairs. Using items with for is 
often the preferred technique for iterating over a dictionary as it gives you access to the 
key and the value as loop variables, which you can then use in your suite. The resulting 
suite is easier on the eye, which makes it easier to read. 


Here is the items equivalent of the above loop code. Note how there are now two 
loop variables in this version of the code (k and v), and that we continue to use the 
sorted function to control the output ordering: 




The W *»W 

wethod yasses 
bafck buo loo\> 
vav-iables. 


»> for k, v in sorted (found. items () ) 



print(k, 'was found 1 , v, f time(s). ! ) 


a 

was 

found 

0 

time(s) 

e 

was 

found 

2 

time (s) 

■ 

i 

was 

found 

0 

time (s) 

o 

was 

found 

0 

time(s) 

u 

was 

found 

0 

time(s) 


as brfove- 


VVc mvoke *the 
hW*’ method 

oy \ -the w ^ou^d w 
die-tiona^Y- 



-•biet "this Code is so 
casicv -to Ytad. 


tWei are no 

Dumb Questipns 


M.- Why are we calling sorted again in the second loop? The first loop arranged the dictionary in the ordering we want, so this 
must mean we don’t have to sort it a second time, right? 

No, not quite. The sorted built-in function doesn’t change the ordering of the data you provide to it, but instead returns an ordered 
copy of the data. In the case of the found dictionary, this is an ordered copy of each key/value pair, with the key being used to determine 
the ordering (alphabetical, from Athrough Z). The original ordering of the dictionary remains intact, which means every time we need to iterate 
over the key/value pairs in some specific order, we need to call sorted, as the random ordering stili exists in the dictionary. 
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structured data 



Frecruency Count 

Having concluded our experimentation at the >>> prompt, it's now time 
to make changes to the vowels3 . py program. Below are all of the code 
snippets we think you might need. Your job is to rearrange the magnets to 
produce a working program that, when given a word, produces a frequency 
count for each vowel found. 



vowels = ['a', 'e', 'i', 'o', 'u'] 

word = input("Provide a word to search for vowels: ") 

Pedidc wVu£V> dodc 

3 ocs m 

o-C -tv>e dashed-tae 
lotatioxs tv-ea-te 



for letter in word: 

if letter in vowels: 


for 


in sorted( 


) : 


print( 


'was found', , 'time(s).') 


do all 

y>? Bc tavc-fub y\oi —^ 
all -these wa^c*b 
needed- 






Once youVe placed the magnets where you think they should go, bring 
vowels3 . py into IDLE's edit window, rename it vowels4 .py, and then 
apply your code changes to the new version of this program. 


you are here ► 
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how many vowels 



Frecruency Count Magnets olui 


ion 


Having concluded our experimentation at the >>> prompt, it was time to 
make changes to the vowels3 . py program. Your job was to rearrange the 
magnets to produce a working program that, when given a word, produces 
a frequency count for each vowel found. 


Once you'd placed the magnets where you thought they should go, 
you were to bring vowels3 . py into an IDLE's edit window, rename it 
vowels4 . py, and then apply your code changes to the new version of 
this program. 


TVis is -the 
vowels^tpy 
procyrar*. 


vowels = ['a', 'e', 'i', ' o', ' u'] 


Cveate a* er*pty 

didtionav-y- V 


word = input("Provide a word to search for vowels: 


found = {} 



nitializje "the value 
assodiated with eadh 
o( the keys (eadh 
vov/el) to 0 . 


As the w WMoop ^ 
is usi»;} the u '»tc^s 
wxcthod) v/e need 
to pvovide two loop 
vanables, 
the key a»d 
the value. 


V £ 


ov 


V* -Cov 




found['a'] =0 
found['e'] =0 
found['i'] =0 
found['o'] =0 
found['u'] =0 


for letter in word: 

if letter in vowels 


/hd\remeht the value 
ve-fevved io by 
•fouhdfle-t-tev-J ' by < 




print( 



in sorted( 


, 'was found', 





Tbe key a*>d tbe value 
ave used "fco eveate 
eatb ou-tyvt ««essa^e- 



) : 


, 'time(s).') 


Imo ke the 
'tems method 
on the w *fouhd w 
didtioha^y to 

addess eaeh vow 
o-P daia with 
caeh iteratio*. 


These »*ay\ets 
were*t *eeded- 



r 
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structured data 



Tqst Drwq 


Let’s take vowels4 .py for a spin. 
performs: 


your code in an IDLE edit window, press F5 to see how it 


vowels4.py - /Users/Paul/Desktop/NewBook/ch02/vowels4.py (3.4.3) 


The w voweU^rpy- 
eode 




V\fe va* *fche tod e *thvee 
Wes “to see hov/ 
pev^fovws* 



vowels = ['a', 'e', 'i', 'o', 'u'] 

word = input( "Provide a word to search for vowels: ") 


found = {} 


found[ 
found[ 
found[ 
found[ 
found[ 


a 

e 

• 

i 

o 

u 


= 0 
= 0 
= 0 
= 0 
= 0 


for letter in word: 

if letter in vowels 
found[letter] + 


= 1 


i 


for k, v in sorted (found.items()): 
print(k, 'was found', v, 'time(s 

Python 3.4.3 Shell 


)-*) 


»> ============== 

»> 

Provide a word to 
a was found 0 time 
e was found 1 time 
i was found 2 time 
o was found 0 time 
u was found 0 time 
»> ============== 

»> 

Provide a word to 
a was found 1 time 
e was found 6 time 
i was found 3 time 
o was found 0 time 
u was found 1 time 
»> ============== 

»> 

Provide a word to 
a was found 0 time 
e was found 0 time 
i was found 0 time 
o was found 0 time 
u was found 0 time 
»> 


RESTART 


search 
(s). 
(s). 
(s). 
(s). 
(s). 


for vowels: hitch-hiker 



RESTART 


search 
(s). 
(s). 
(s). 
(s). 
(s). 


for vowels: life, the universe, and everything 

These ihiree Vuhs w 


RESTART 


search 
(s). 
(s). 
(s). 
(s). 
(s). 


for vowels: sky 



pv*odu£e -the ouipui we 
ex feti -the* io. 


O 




you are here ► 
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no more zeros 


Just How Pynamic Are Pictionaries? 

The vowels4 . py program reports on all the found vowels, even when they 
arenl found. This may not bother you, but let’s imagine that it does and you 
want this code to only display results when results are actually found. That is, 
you don’t want to see any of those “found 0 time(s)” messages. 

How might you go about solving this problem? 



Py+hon's dic+ionary is dynamic, right? So, 
all we have to do is remove those fi ve lines 
that initialize each vowel's frequency count? 
With those lines gone, only found vowels will be 
counted, right? 


That sounds like it might work. 

We currently have five lines of code near 
the start of the vowels4 . py program 
that weVe included in order to initially set 
each vowels frequency count to 0. This 
creates a key/value pair for each vowel, even 
though some may never be used. If we take 
those five lines away, we should end up only 
recording frequency counts for found vowels, 
and ignore the rest. 

Let’s give this idea a try. 


This is -the 'vowels^. 

tode v/rth -the mrtialiaa-tioh 
tode v-emoved. 



Jjo thlsf 





vowels5.py - /Users/Paul/Desktop/_NewBook/ch03/vowels5.py (3.4.3) 


Take the code in vowels4 .py 
and save it as vowe 1 s 5 . py. 
Then remove the five lines of 
initialization code. Your IDLE 
edit window should look like 
that on the right of this page. 


_ . i i i i i • i i i i i 


vowels = ['a , ’e' , 'i', ’ o' , 'u'] 

word = input( Provide a word to search for vowels: " ) 


found = {} 

for letter in word: 

if letter in vowels: 
found[letter] += 1 

for k, v in sorted (found.items()): 

print(k, 'was found', v, time(s). ) 
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Ln: 13 Coi: 0 


















structured data 



You know the drill. Make sure vowels5. py is in an IDLE edit window, then press F5 to run your 
program. You’11 be confronted by a runtime error message: 


Python 3.4.3 Shell 



RESTART 


»> ================================ 

»> 

Provide a word to search for vowels: hitchhiker 
Traceback (most recent call last): 

File "/Users/Paul/Desktop/_NewBook/ch03/vowels5.py", line 9, in <module> 
found[letter] += 1 
KeyErrors 'i' 

l»> 


Ln: 11 Coi: 0 


Ttos ta* *t 

be $ood- 


lt’s ciear that removing the five lines of initialization code wasn’t the way to go here. But why has this 
happened? The fact that Python’s dictionary grows dynamically at runtime should mean that this code 
cannot crash, but it does. Why are we getting this error? 


Pictionary keys must be mitialized 

Removing the initialization code has resulted in a runtime error, specifically 
a KeyError, which is raised when you try to access a value associated with 
a nonexistent key. Because the key can’t be found, the value associated with it 
cani be found either, and you get an error. 

Does this mean that we have to put the initialization code back in? After ali, it 
is only five short lines of code, so what 5 s the harm? We can certainly do this, 
but leti think about doing so for a moment. 

Imagine that, instead of five frequency counts, you have a requirement to 
track a thousand (or more). Suddenly, we have lots of initialization code. We 
could “automate” the initialization with a loop, but we’d stili be creating a 
large dictionary with lots of rows, many of which may end up never being 
used. 


Gee]c BffS i 


An alternative approach to 
handling this issue is to deal 
with the run-time exception 
raised here (which is a 
"KeyError" in this example). 
We're holding off talking 
about how Python handles 
run-time exceptions until a 
later chapter, so bear with 
us for now. 



If only there were a way to create a key/value pair on the fly, just as soon as 
we realize we need it. 



I wonder does the "in" 
operator work with 
dictionaries? 


That’s a great question. 

We first met in when checking lists for a value. 
Maybe in works with dictionaries, too? 

Leti experiment at the >>> prompt to find out. 


you are here ► 
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check with in 


Avoidiwg KeyErrors at Ruwtiwe 

As with lists, it is possible to use the in operator to check whether a key exists in a 
dictionary; the interpreter returns True or False depending on what’s found. 

Let 5 s use this fact to avoid that KeyError exception, because it can be annoying 
when your code stops as a resuit of this error being raised during an attempt to 
populate a dictionary at runtime. 


key#4 

object 

key#l 

object 

key#3 

object 

key#2 

object 


Dictionary 


To demonstrate this technique, we’re going to create a dictionary called f ruits, 
then use the in operator to avoid raising a KeyError when accessing a 
nonexistent key We start by creating an empty dictionary; then we assign a 
key/value pair that associates the value 10 with the key apples. With the row 
of data in the dictionary we can use the in operator to confirm that the key 
apples now exists: 


»> f ruits 


{> 

»> fruits [' apples ' ] = 10 
»> fruits 
{'apples': 10} 

»> ' apples' in fruits 



This is ali as e% ? etied. The value 
•s assodiated with the key, and 
theire s no «runtime error when we 
we the^ in operator to ehedk -for 

the key s exis-fcenee. 


True 



Before we do anything else, let’s consider how the interpreter views the fruits 
dictionary in memory after exeeuting the above code: 


fruits 



The “apples" key 
is assoeiated with 
"the value 10. 


tliere icire no 

Dumb Qiiestions 


O: I take it from the example on this page that Python uses the constant value True for true ? Is there a False, too, and does 
case matter when using either of these values? 


Yes, to all those questions. When you need to specify a boolean in Python, you can use either True or False. These are constant 
values provided by the interpreter, and must be specified with a leading uppercase letter, as the interpreter treats true and false as 
variable names, not boolean values, so care is needed here. 
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structured data 


Checking for Membership with "in" 

Let 5 s add in another row of data to the f ruits dictionary for bananas and 
see what happens. However, instead of a straight assignment to bananas, 

(as was the case with apples), let 5 s increment the value associated with 
bananas by 1 if it already exists in the f ruits dictionary or, if it doesn’t 
exist, let’s initialize bananas to 1. This is a very common activity, especially 
when you’re performing frequency counts using a dictionary, and the logic we 
employ should hopefully help us avoid a KeyError. 


key#4 

object 

key#l 

object 

key#3 

object 

key#2 

object 


Dictionary 





In the code that follows, the in operator in conjunction with an if statement 
avoids any slip-ups with bananas, which—as wordplays go—is pretty bad 
(even for us): 


Be-fove the 
bananas todt vuns 


»> if 1 bananas 1 in f ruits : 
fruits['bananas'] += 1 


fruits[ 1 bananas 1 ] = 1 
»> f ruits 

{ 1 bananas 1 : 1, 1 apples 1 : 10} 

'f_ 



We've set the "bananas" value to I 


The above code changes the state of the f ruits dictionary within the 
interpretebs memory, as shown here: 


As expected, the f ruits dictionary has grown by one key/value pair, and 
the bananas value has been initialized to 1. This happened because the 
condition associated with the if statement evaluated to False (as the key 
wasn’t found), so the second suite (that is, the one associated with else^ 
exeeuted instead. Let’s see what happens when this code runs again. 


I/Ve dhedk io scc i-f -the “bananas” key 
is in the didtionav-y, and as it isn't, we 
initialize its value io I. Critically, we 
avoid any possibility of a "KeyError". 





If you are familiar with the ? : 
ternary operator from other 
languages, note that Python 
supports a similar construet. You 
can say this: 

x = 10 if y > 3 else 20 

to set x to either 10 or 20 
depending on whether or not the 
value of y is greater than 3. That 
said, most Python programmers 
frown on its use, as the equivalent 
if . . . else. . . statements 
are considered easier to read. 
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one more time 


Ensuring Imtializatiow Pefore Use 

If we execute the code again, the value associated with bananas should now 
be increased by 1, as the i f suite executes this time due to the fact that the 
bananas key already exists in the f ruits dictionary: 



key#4 

object 

key#l 

object 

key#3 

object 

key#2 

object 


Dictionary 


BeW -the "banahas" 
£ode \ruhs (agam) 


To run this code again, press Ctrl-P (on a Mac) or Alt-P (on Linux/Windows) to 
cycle back through your previously entered code statements while at IDLE 5 s >>> 
prompt (as using the up arrow to recall input doesn’t work at IDLE’s >>> prompt). 
Remember to press Enter twice to execute the code once more: 


»> if 1 bananas 1 in f ruits: 
fruits[ 1 bananas'] += 1 


fruits[ 1 bananas 1 ] = 1 


»> f ruits 

{ 1 bananas 1 : 2, 1 apples 1 : 10} 

- Hfe'* irtveased the 'Was" 
value by I. 



This time avouhd, the “balabas” key 
does exist ib the dictioba\ry, so y/e 
ibe\remebt its value by I. fys be-Pove, 
ouv- use o-P i-P abd w ib W togethev 
stop a u ^eyE\nro\r w exceptiob -Pvom 
Crashibg this Code* 


As the code associated with the i f statement now executes, the value associated 
with bananas is incremented within the interpreter’s memory: 



A-ftev- the u bab3bas 
Code vubs, the value 
associated with 
w bababas w has ibCv-eased- 


This mechanism is so common that many Python programmers shorten these four 
lines of code by inverting the condition. Instead of checking with in, they use 
not in. This allows you to initialize the key to a starter value (usually 0) if it isn’t 
found, then perform the increment right after. 

Let’s take a look at how this mechanism works. 
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structured data 


Substitutiwg "wot iw" for "m" 


At the bottom of the last page, we stated that most Python programmers refactor 
the original four lines of code to use not in instead of in. Let’s see this in 
action by using this mechanism to ensure the pe ars key is set to 0 before we try 
to increment its value: 


»> if f pears f not in fruits: 

fruits['pears'] = 0 


fnrtidlize (it 


key#4 

object 

key#l 

object 

key#3 

object 

key#2 

object 


Dictionary 

needed). 


»> fruits [ 'pears ' ] += 1 __ |ndveme»-l- 

»> fruits 

{'bananas': 2, 'pears': 1, 'apples': 10} 

These three lines of code have grown the dictionary once more. There are now 
three key/value pairs in the fruits dictionary: 



A-ftev Ue -Uo 

P U ^ 

Imes ok \>cav-s 
todt v-uns 


The above three lines of code are so common in Python that the language 
provides a dictionary method that makes this if/not in combination more 
convenient and less error prone. The setdef au it method does what the two- 
line if/not in statements do, but uses only a single line of code. 


Here’s the equivalent of the pears code from the top of the page rewritten to 
use setdefault: 


»> fruits . setdefault ('pears ' , 0) \_(hi-tializj; (i-f ntedtA). 

»> fruits [ 'pears ' ] += 1 
»> fruits 

{'bananas': 2, 'pears': 2, 'apples': 10} ' x - 


The single call to setdetf au it has replaced the two-line 
if/not in statement, and its usage guarantees that a key 
is always initialized to a starter value before it’s used. Any 
possibility of a KeyError exception is negated. The current 
state of the fruits dictionary is shown here (on the right) to 
confirm that invoking setdefault after a key already exists 
has no effect (as is the case with pears), which is exactly 
what we want in this case. 


fruits 
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long live setdefault 


Puttiwg the "setdefault" Method to Work 

Recall that our current version of vowels5 . py results in a runtime error, 
specifically a KeyError, which is raised due to our code trying to access the 
value of a nonexistent key: 


# # # vowels5.py - /Users/Paul/Desktop/NewBook/ch03/vowels5.py (3.4.3) 


key#4 

object 

key#l 

object 

key#3 

object 

key#2 

object 


r 

TKis dode 
pvodudes 
■tKis cvvov. 



vowels = ['a', 'e', 'i*, 'o', 'u'] 

word = input( Provide a word to search for vowels: " ) 

found = {} 

for letter in word: 

if letter in vowels: 
found[letter] += 1 

for k, v in sorted (found.items()): 

print(k, 'was found', v, time(s). ) 

Python 3.4.3 Shell 


Dictionary 


RESTART 


»> ================================ 

»> 

Provide a word to search for vowels: hitchhiker 
Traceback (most recent call last): 

File "/Users/Paul/Desktop/_NewBook/ch03/vowels5.py", line 9, in <module> 
found[letter] += 1 
KeyError: 'i' 

l»> 


Ln: 11 Coi: 0 


From our experiments with fruits, we know we can call setdefault 
as often as we like without having to worry about any nasty errors. We 
know setdefault’s behavior is guaranteed to initialize a nonexistent key 
to a supplied default value, or to do nothing (that is, to leave any existing 
value associated with any existing key alone). If we invoke setdefault 
immediately before we try to use a key in our vowels5 . py code, we are 
guaranteed to avoid a KeyError, as the key will either exist or it won’t. 
Either way, our program keeps running and no longer crashes (thanks to our 
use of setdefault). 

Within your IDLE edit window, change the first of the vowels5 . py 
program’s for loops to look like this (by adding the call to setdefault), 
then save your new version as vowels 6 . py: 


Use "setdelault” 
to help avoid 
the "KeyError” 

excepi ion. 


for letter in word: 

if letter in vowels: 

found.setdefault(letter, 
found[letter] += 1 


0 ) 



sm^le Ime <£ to&c 

da* o£*tc* all 
-the diWerenfce. 
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structured data 



With the most recent vowels6 .py program in your IDLE edit window, press F5. Run this version a 
few times to confirm the nasty KeyError exception no longer appears. 




Python 3.4.3 Shell 


RESTART 


»> 

»> 

Provide a word to search for vowels: hitch-hiker 
e was found 1 time(s). 
i was found 2 time(s). 

»> ================================ RESTART =========================== 

»> 

Provide a word to search for vowels: life, the universe, and everything 
a was found 1 time(s). 
e was found 6 time(s). 
i was found 3 time(s). 
u was found 1 time(s). 

»> 


The use of the setdef ault method has solved the KeyError problem 
we had with our code. Using this technique allows you to dynamically grow a 
dictionary at runtime, safe in the knowledge that you’11 only ever create a new 
key/value pair when you actually need one. 



Ln: 23 Coi: 4 


This is lookmg good. The 
KeyEvvoir” is gohe. 


When you use setdef ault in this way, you never need to spend time 
initializing all your rows of dictionary data ahead of time. 

Pictionaries: updatiwg what we already kwow 


Let 5 s add to the list of things you now know about Python’s dictionary: 



BULLET POINTS 


■ By default, every dictionary is unordered, as insertion 
order is not maintained. If you need to sort a dictionary 
on output, use the sorted built-in function. 

■ The items method allows you to iterate over a 
dictionary by row—that is, by key/value pair. On each 
iteration, the items method returns the next key and 
its associated value to your for loop. 


Trying to access a nonexistent key in an existing 
dictionary results in a KeyError. When a 
KeyError occurs, your program crashes with a 
runtime error. 

You can avoid a KeyError by ensuring every key 
in your dictionary has a value associated with it before 
you try to access it. Although the in and not in 
operators can help here, the established technique is to 
use the setdef ault method instead. 
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how much more? 


Aren't Pictionaries (and Lists) Enough? 



Weve been talking about data structures 
for ages...how much more of this is there? 
Surely dictionaries—together with lists— 
are ali I'll need most of the time? 


Dictionaries (and lists) are great. 

But they are not the only show in town. 

Granted, you can do a lot with dictionaries and 
lists, and many Python programmers rarely 
need anything more. But, if truth be told, these 
programmers are missing out, as the two remaining 
built-in data structures—set and tuple—are useful 
in s pecific circumstances , and using them can greatly 
simplify your code, again in specific circumstances. 

The trick is spotting when the specific circumstances 
occur. To help with this, let’s look at typical examples 
for both set and tuple, starting with set. 


-there-, scce no 

Dumb Questipns 


Is that it for dictionaries? Surely it’s common for the value part of a dictionary to be, for instance, a list or another dictionary? 

Yes, that is a common usage. But we’re going to hang on until the end of this chapter to show you how to do this. In the meantime, let 
what you already know about dictionaries sink in... 
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structured data 


Sets Pont Allow Puplicates 

Python’s set data structure is just like the sets you learned about in school: it has 
certain mathematical properties that always hold, the key characteristic being that 
duplicate values are forbidden. 

Imagine you are provided with a long list of all the first names for everyone in a large 
organization, but you are only interested in the (much smaller) list of unique first 
names. You need a quick and foolproof way to remove any duplicates from your long 
list of names. Sets are great at solving this type of problem: simply convert the long 
list of names to a set (which removes the duplicates), then convert the set back to a list 
and—ta da!—you have a list of unique first names. 

Python’s set data structure is optimized for very speedy lookup, which makes using a 
set much faster than its equivalent list when lookup is the primary requirement. As lists 
always perform slow sequential searches, sets should always be preferred for lookup. 



Set 


Spotting sets in your code 


Sets are easy to spot in code: a collection of objects are separated from one another by 
commas and surrounded by curly braces. 


For example, here 5 s a set of vowels: 


»> vowels 
»> vowels 

Chedk oul the ov-dev»^- t ' e ' ' ' u ' / 

It’s dhan^ed tv-om what v/as ^ 

ov-icynally msertedi and the 
deitates are <y>*e t oo- 



Sets start and end with a durly brate- 


= { 'a', 'e', 'e', 'i', 'o', 'u', 'u' } 


'a', 'i', 'o'} 



^bjedts are separated 
Wom one dho-fchev- by a 


The fact that a set is enclosed in curly braces can often resuit in your brain mistaking a 
set for a dictionary, which is also enclosed in curly braces. The key difference is the use 
of the colon character (:) in dictionaries to separate keys from values. The colon never 
appears in a set, only commas. 

In addition to forbidding duplicates, note that—as in a dictionary—insertion order 
is not maintained by the interpreter when a set is used. However. like all other data 
structures, sets can be ordered on output with the sorted function. And, like lists and 
dictionaries, sets can also grow and shrink as needed. 

Being a set, this data structure can perform set-like operations, such as difference , 
intersection, and union. To demonstrate sets in action, we are going to revisit our vowel 
counting program from earlier in this chapter once more. We made a promise when 
we were first developing vowels3 . py (in the last chapter) that we’d consider a set 
over a list as the primary data structure for that program. Let 5 s make good on that 
promise now. 
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sets hate duplicates 


Creating Sets Efficiently 

Let 5 s take yet another look at vowels3 . py, which uses a list to work out which 
vowels appear in any word. 

Here’s the code once more. Note how we have logic in this program to ensure we 
only remember each found vowel once. That is, we are very deliberately ensuring 
that no duplicate vowels are ever added to the found list: 


objecta ob J ecte 

objectf ° bjeCtd 

object c ° bj6Ct b 

Se t 


This is 

(M VC^OV“ts OY\ 

-the utti<\uC vowels -^ 

•(•ouiftdi in a v/ovd- 
This Lo&t uses 3 list 
as its pri^av-y data 
stvuctuv-e. 


# # # vowels3.py - /Users/Paul/DesktopONewBook/ch02/vowels3.py (3.4.3) 

vowels = [ 'a' , 'e', 'i', 'o', 'u'] 

word = input( "Provide a word to search for vowels: ) 

found = [] 

for letter in word: 

if letter in vowels: 

if letter not in found: ^^ 

found append(letter) ^ d^teates 

for vowel in found: vvc * „ .. J 

print( vowel) m the tound list- 



Ln: 11 Coi: 0 



Before continuing, use IDLE to save this code as vowels7 . py so that we can 
make changes without having to worry about breaking our list-based solution 
(which we know works). As is becoming our Standard practice, let’s experiment at 
the >>> prompt first before adjusting the vowels7 . py code. We’ll edit the code 
in the IDLE edit window once weVe worked out the code we need. 

Creating sets from sequences 

We start by creating a set of vowels using the code from the middle of the last 
page (you can skip this step if youVe already typed that code into your >>> 
prompt): 


»> vowels ={ ' a' , ' e' , ' e' , ' i' , 1 o 1 f 

»> vowels 

{ f e f , ! u f , ! a ! , ' i' , ! o f } 

Below is a useful shorthand that allows you to pass any sequence (such as a string) 
to the set function to quickly generate a set. Here’s how to create the set of 
vowels using the set function: 



These two |j hCS 
do the sar*e thihg: both 
dss *9 h a hCW sc -(. 0 h; c ^ ^ 
a vanable. 


»> vowels2 = set ( 1 aeeiouu 1 ) 
»> vowels2 

{ ! e ! , ! u ! , ' a' , ! i ! , ! o ! } 
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structured data 


Takiwg Advantage of Set Methods 

Now that we have our vowels in a set, our next step is to take a word and 
determine whether any of the letters in the word are vowels. We could do this by 
checking whether each letter in the word is in the set, as the in operator works 
with sets in much the same way as it does with dictionaries and lists. That is, 
we could use in to determine whether a set contains any letter, and then cycle 
through the letters in the word using a for loop. 

However, let’s not follow that strategy here, as the set methods can do a lot of this 
looping work for us. 

There 5 s a much better way to perform this type of operation when using sets. It 
involves taking advantage of the methods that come with every set, and that 
allow you to perform operations such as union, difference, and intersection. Prior 
to changing the code in vowels7 . py, let 5 s learn how these methods work by 
experimenting at the >>> prompt and considering how the interpreter sees the 
set data. Be sure to follow along on your computer. Let’s start by creating a set of 
vowels, then assigning a value to the word variable: 


objecta ob j eCte 


object f 


object 


object d 
object b 


■> 

■\ 


Set 


»> vowels = set (' aeiou 1 ) 
»> word = ! hello ! 


The interpreter creates two objects: one set and one string. Here 5 s what the 
vowels set looks like in the interpreter’s memory: 


vowels 



The set 
£oivfcaihs -the 

tive le-fctev 



Let 5 s see what happens when we perform a union of the vowels set and the set 
of letters created from the value in the word variable. We’ll create a second set 
on-the-fly by passing the word variable to the set function, which is then passed 
to the union method provided by vowels. The resuit of this call is another 
set, which we assign to another variable (called u here). This new variable is a 
combination of the objects in both sets (a union): 

»> u = vowels .union ( 



set(word) ) 


Py-thoh £ohve\rs-b -the value 
,h "woird” ih-to a se-t ot 
lette\r objects 6rer*ovihg any 
dupliea-tes as it does so). 


The union method £or*bihes 
ohe set with aether, whith 
is the* assi^ed to a *ew 
variable talled V (whidh is 
aether set). 



After this call to the union method, what do 
the vowels and u sets look like? 


you are here ► 


125 









































fun with sets 


union Works by Cowbining Sets 

At the bottom of the previous page we used the union method to create a 
new set called u, which was a combination of the letters in the vowels set 
together with the set of unique letters in word. The act of creating this new 
set has no impact on vowels, which remains as it was before the union. 
However, the u set is new, as it is created as a resuit of the union. 


objecta ob j eCte 


object f 


object 


object d 
object b 


"N 

■\ 


Set 



What happewed to the loop code? 


That single line of code packs a lot of punch. Note that you haven’t 
specifically instructed the interpreter to perform a loop. Instead, you told the 
interpreter what you wanted done—not how you wanted it done—and the 
interpreter has obliged by creating a new set containing the objects you’re 
after. 


A common requirement (now that weVe created the union) is to turn the 
resulting set into a sorted list. Doing so is trivial, thanks to the sorted and 
list functions: 

»> u_list = sorted (list (u) ) 
»> u_list 

[ ! a ! , ! e ! , f h ! , ' i' , ! l f , ! o ! , 


A sov-fced lis-fc o-P 

ubique lettev-s 
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differewce Telis You WhaTs Not Shared 

Another set method is dif ference, which, given two sets, can teli you 
what’s in one set but not the other. Let’s use dif ference in much the same 
way as we did with union and see what we end up with: 

»> d = vowels . dif ference (set (word)) 
»> d 

{'u', 'i', 'a'} 


objecta ob j eCte 


object f 


object 


object d 
object b 


■> 

■\ 


Set 


The dif ference function compares the objects in vowels against the 
objects in set (word), then returns a new set of objects (called d here) 
which are in the vowels set but not in set (word) . 

Here’s what happens: 



We once again draw your attention to the fact that this outcome has been 
accomplished without using a for loop. The difference function does all 
the grunt work here; all we did was state what was required. 

Flip over to the next page to look at one final set method: intersection. 
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what is shared 


intersection Reports on Cowwonality 

The third set method that we’ll look at is intersection, which takes the 
objects in one set and compares them to those in another, then reports on any 
common objects found. 

In relation to the requirements that we have with vowel s 7 . py, what the 
intersection method does sounds very promising, as we want to know which 
of the letters in the user’s word are vowels. 

Recall that we have the string "heiio” in the word variable, and our vowels in 
the vowels set. Here’s the intersection method in action: 


objecta ob j eCte 


object f 


object 


object d 
object b 


“\ 


Set 


»> i = vowels . intersection (set (word) ) 
»> i 
{ f e f , f o ! } 

The intersection method confirms the vowels e and o are in the word 
variable. Here’s what happens: 


The set of vov/els 


The word u hello w is 
■fcuvhed inio a set. 




set(word) 


i = vowels.intersection(set(word)) 




V set fccwsists of ali 
obietts m w voY/eU w that 
also m u set(wo\rd) ,, « 


There are more set methods than the three weVe looked at over these last few 
pages, but of the three, intersection is of most interest to us here. In a single 
line of code, weVe solved the problem we posed near the start of the last chapter: 
identijy the vowels in any string. And all without having to use any loop code. Let 5 s 
return to the vowels7 . py program and apply what we know now. 
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structured data 


Sets: What You Already Kwow 

Here’s a quick rundown of what you already know about Python’s set data 
structure: 



BULLET POINTS 


■ Sets in Python do not allow duplicates. 

■ Like dictionaries, sets are enclosed in curly braces, 
but sets do not identify key/value pairs. Instead, each 
unique object in the set is separated from the next by a 
comma. 


■ Also like dictionaries, sets do not maintain insertion 
order (but can be ordered with the sorted function). 


You can pass any sequence to the set function 
to create a set of elements from the objects in the 
sequence (minus any duplicates). 

Sets come pre-packaged with lots of built-in functionality, 
including methods to perform union, difference, and 
intersection. 



Here is the code to the vowels3 . py program once more. 

Based on what you now know about sets, grab your pencil and 
strike out the code you no longer need. In the space provided on 
the right, provide the code you'd add to convert this list-using 
program to take advantage of a set. 

Hint: you'll end up with a lot less code. 


vowels = ['a', ' e', 'i’, 'o', ’ u'] 

word = input("Provide a word to search 

found = [] 

for letter in word: 

if letter in vowels: 

if letter not in found: 
found.append(letter) 
for vowel in found: 
print(vowel) 


for vowels: ") 


When youfe done, be sure to rename your file vowels7 . py. 
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vowels with sets 


Sharpen your pencil 
^ - Solutiori 



Theres lo*b o£ 
tode *bo yt r*»d of • 


Here is the code to the vowels3 . py program once more. 

Based on what you now know about sets, you were to grab your 
pencil and strike out the code you no longer needed. In the space 
provided on the right, you were to provide the code you'd add to 
convert this list-using program to take advantage of a set. 


Hint: you'll end up with a lot less code. 


4 / 


vowels =■ setCaeiou) 



Create a S et 
o-P vowels. 


word = input("Provide a word to search for vowels: ") 



When you were done, you were to rename your file vowels7 . py. 



I feel cheated...all that time wasted 
learning about lists and dictionaries, and 
the best solution to this vowels problem 
all along was to use a set? Seriously? 


It wasn’t a waste of time. 

Being able to spot when to use one built-in data 
structure over another is important (as you’11 want to 
be sure you’re picking the right one). The only way you 
can do this is to get experience using all of them. None 
of the built-in data structures qualify as a “one size 
fits alf 5 technology, as they all have their strengths and 
weaknesses. Once you understand what these are, you’11 
be better equipped to select the correct data structure 
based on your application’s specific data requirements. 
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structured data 



Let’s take vowels7 .py for a spin to confirm that the set-based version of our program runs as 
expected: 


Our latest tode 



# # # vowels7.py - /Users/Paul/Desktop/NewBook/ch03/vowels7.py (3.4.3) 


vowels = set(aeiou ) 

word = input ( "Provide a word to search for vowels: ) 
found = vowels.intersection( set (word)) 
for vowel in found: 
print (vowel) 


• O • Python 3.4.3 Shell 



dictionary works best. However, if you are more concerned with maintaining 
insertion order, then only a list will do...which is almost true. There 5 s one 
other built-in data structure that maintains insertion order, and which weVe 


yet to discuss: the tuple. 


Let’s spend the remainder of this chapter in the company of Python’s tuple. 


you are here ► 
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why? 


Making the Case for Tuples 

When most programmers new to Python first come across the tuple, they 
question why such a data structure even exists. After ali, a tuple is like a list 
that cannot be changed once it’s created (and populated with data). Tuples 
are immutable: they cannot change. So, why do we need them? 



Tuple 


It turns out that having an immutable data structure can often be useful. 
Imagine that you need to guard against side effects by ensuring some data 
in your program never changes. Or perhaps you have a large constant list 
(which you know won’t change) and you’re worried about performance. 
Why incur the cost of all that extra (mutable) list processing code if you’re 
never going to need it? Using a tuple in these cases avoids unnecessary 
overhead and guards against nasty data side effects (were they to occur). 

How to spot a tuple in code 

As tuples are closely related to lists, it’s no surprise that they look similar 
(and behave in a similar way. Tuples are surrounded by parentheses, 
whereas lists use square brackets. A quick visit to the >>> prompt lets us 
compare tuples with lists. Note how we’re using the type built-in function 
to confirm the type of each object created: 


tliereiare no 

Diimb QuestiQns 



Where does the name “tuple” come from? 


It depends whom you ask, but the name has 
its origin in mathematics. Find out more than you’d 
ever want to know by visiting https.Y/en.wikipedia. 
org/wiki/Tuple. 


Thev-es «otV»* (»> vowels = [ 'a', 'e', 'i', 

new here A list > -i »> type (vowels) 
vowels is Created- L <class 'list'> 



Now that vowels and vowels2 exist (and are populated with data), we 
can ask the shell to display what they contain. Doing so confirms that the 
tuple is not quite the same as the list: 



'U' ] 



f U f ) -^ 

This tuple looks 
l'ke a list, but 
ish t- Tuples are 
suvvouhded by 
parentheses (not 
square brackets). 


»> vowels 

['a', 'e', 'i', 'o', 'u'] 

»> vowels2 

('a', 'e', 'i', 'o', 'u') 4 

But what happens if we try to change a tuple? 


The 

parentheses 
'"dicate that 

this is a tuple. 
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structured data 


Tuples Are Immutable 

As tuples are sort of like lists, they support the same square bracket notation 
commonly associated with lists. We already know that we can use this 
notation to change the contents of a list. Here’s what we’d do to change the 
lowercase letter i in the vowels list to be an uppercase I: 



Tuple 


»> vowels 
»> vowels 
[' a ! , 'e' 


[2] = ! I ! 



Assigh dh uppcv^asc T ; 

ot -the vowels" lis-fc. 


As expected, the third element in the list (at index location 2) has changed, 
which is fine and expected, as lists are mutable. However, look what happens 
if we try to do the same thing with the vowels2 tuple: 

»> vowels2[2] = ! I f " 

Traceback (most recent call last): 

File "<pyshell#16 > n , line 1, in <module> v —_ 

vowels2[2] = f I f 

TypeError: 1 tuple 1 object does not support item assignment 
»> vowels2 

(' a f , ! e ! , ! i ! , 'o' , 1 u 1 ) 

Uo ehahge heve, as -tuples 

dire innrKuiable 

Tuples are immutable, so we can’t complain when the interpreter protests 
at our trying to change the objects stored in the tuple. After ali, that 5 s the 
whole point of a tuple: once created and populated with data, a tuple cannot 
change. 



The ih-teirpve-ter 
£or*plaihS loudly i-f you 
^hahje a tuple. 


Make no mistake: this behavior is useful, especially when you need to ensure 
that some data can’t change. The only way to ensure this is to put the data in 
a tuple, which then instructs the interpreter to stop any code from trying to 
change the tuple’s data. 

As we work our way through the rest of this book, we’ll always use tuples 
when it makes sense to do so. With reference to the vowel-processing code, it 
should now be ciear that the vowels data structure should always be stored 
in a tuple as opposed to a list, as it makes no sense to use a mutable data 
structure in this instance (as the five vowels never need to change). 


II the data in 
your structure 
never changes, 
put it in a tuple 




There’s not much else to tuples—think of them as immutable lists, nothing 
more. However, there is one usage that trips up many a programmer, so let’s 
learn what this is so that you can avoid it. 


you are here ► 
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a tuple caveat 


Watch Out for Siwgle-Object Tuples 


Let’s imagine you want to store a single string in a tuple. It 5 s tempting to put 
the string inside parentheses, and then assign it to a variable name...but doing 
so does not produce the expected outcome. 


Take a look at this interaction with the >>> prompt, which demonstrates 
what happens when you do this: 


»> t = (' Python 1 ) 
»> type(t) "N 

<class ! str ! > 1 

»> t \ * 

' Python' J 


This is hot what wc 
expected. Wt\t cndcdt 
up wi-fch a strmg. m>at 
happened io ouv- -tuple? 


What looks like a single-object tuple isn’t; it’s a string. This has happened 
due to a syntactical quirk in the Python language. The rule is that, in order 
for a tuple to be a tuple, every tuple needs to include at least one comma 
between the parentheses, even when the tuple contains a single object. This 
rule means that in order to assign a single object to a tuple (we’re assigning a 
string object in this instance), we need to include the trailing comma, like so: 



Tuple 



»> t2 


( 1 Python 



This looks a little weird, but don’t let that worry you. Just remember this 
rule and you’11 be fine: every tuple needs to include at least one comma between the 
parentheses. When you now ask the interpreter to teli you what type t2 is 
(as well as display its value), you learn that t2 is a tuple, which is what is 
expected: 


That tvdili*^ fcomn>a «*akes 
all the ditw«te, as it 
■telis the 'mtevpvetev that 
this is a tuple- 


»> type (t2) 
<class 'tuple'> 
>» t2 


That’s better: we how 
have a tuple. 


(' Python ' ,) -. TKe interpreter disylays 

the smO|le-°bjett tuple 
with the trailmo, tomma- 

It is quite common for functions to both accept and return their arguments 
as a tuple, even when they accept or return a single object. Gonsequently, 
yomll come across this syntax often when working with functions. We’ll have 
more to say about the relationship between functions and tuples in a little bit; 
in fact, we’11 devote the next chapter to functions (so you won’t have long to 
wait). 



Now that you know about the four data structure built-ins, and before we get 
to the chapter on functions, let’s take a little detour and squeeze in a short— 
and fun!—example of a more complex data structure. 
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structured data 


Combining the Puilt-in Pata Structores 


o 


AII this talk of data structures 
has me wondering if things can get 
more complex. Specif ically, can I 
store a dic+ionary in a dictionary? 



This question gets asked a lot. 

Once programmers become used to storing numbers, 
strings, and booleans in lists and dictionaries, they very 
quickly graduate to wondering whether the built-ins 
support storing more complex data. That is, can the 
built-in data structures themselves store built-in data 
structures? 

The answer is yes, and the reason this is so is due to 
the fact that everything is an object in Python. 

Everything we Ve stored so far in each of the built-ins 
has been an object. The fact they Ve been “simple 
objects” (like numbers and strings) does not matter, as 
the built-ins can store any object. All of the built-ins 
(despite being “complex”) are objects, too, so you can 
mix-and-match in whatever way you choose. Simply 
assign the built-in data structure as you would a simple 
object, and you’re golden. 

Let’s look at an example that uses a dictionary of 
dictionaries. 


thereifire no 

- Dumb Questipns - 

Does what yoifre about to do only work with dictionaries? Can I have a list of lists, or a set of lists, or a tuple of dictionaries? 

Yes, you can. We’ll demonstrate how a dictionary of dictionaries works, but you can combine the built-ins in whichever way you choose. 


you are here ► 


135 






a mutable table 


Storiwg a Table of Pata 

As everything is an object, any of the built-in data structures can be stored in any 
other built-in data structure, enabling the construction of arbitrarily complex data 
structures...subject to your brain’s ability to actually visualize what’s going on. For 
instance, although a dictionary of lists containing tuples that contain sets of dictionaries 
might sound like a good idea, it may not be, as its complexity is off the scale. 

A complex structure that comes up a lot is a dictionary of dictionaries. This 
structure can be used to create a mutable table. To illustrate, imagine we have this 
table describing a motley collection of characters: 


Name 

Gender 

Occupation 

Home Planet 

Ford Pretedt 

/Hale 

Researcher 

Betelgeuse Seven 

Afthuv* Dent 

/Hale 

Sandwidh-Makev- 

Eavth 

Tvidia /kldMillan 

Fcmalc 

Matke**atidian 

Eavth 

Mav-vin 

Unknown 

Pavanoid A^dv-oid 

Unknown 


Recall how, at the start of this chapter, we created a dictionary called 
person3 to store Ford PrefecFs data: 


i 


) 


person3 = { 'Name': 'Ford Prefect', 

'Gender': 'Male', 

'Occupation': 'Researcher', 

1 Home Planet 1 : 'Betelgeuse Seven' } 


Rather than create (and then grapple with) four individual dictionary 
variables for each line of data in our table, let’s create a single dictionary 
variable, called people. We’ll then use people to store any number of 
other dictionaries. 


To get going, we first create an empty people dictionary, then assign Ford 
Prefect’s data to a key: 

S-tav-t with a new, empty didtionav-y. 

»> people = {} 

»> people [' Ford' ] = { 'Name': 'Ford Prefect', 

f' ' Gender' : ' Male' , 

The key ' s ro™ » _ ' Occupa tion' : ' Researcher' , 

and the 1 i »< ■ Home Planet ' : ' Betelgeuse Seven ' 

anothev dittionavy- 


} 
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A Pictionary Containing a Pictionary 

With the pe opi e dictionary created and one row of data added (Ford’s), we can 
ask the interpreter to display the pe opi e dictionary at the >>> prompt. The 
resulting output looks a little confusing, but all of our data is there: 


A dictionary embedded 

' h a dictionary—note 

the extra curly braces. 


»> people 

{'Ford': {'Occupation': 'Researcher', 'Gender': 'Male', 

'Home Planet': 'Betelgeuse Seven', 'Name': 'Ford Prefect'}} 



There is only one embedded dictionary in people (at the moment), so calling 
this a “dictionary of dictionaries” is a bit of a stretch, as people contains just 
the one right now. Here’s what people looks like to the interpreter: 


people 



The yo?le” 

di£.t>or>avy— 


■■■CohiaiM another dictionary 

(whieh is the value associated_ 

'with the "Ford" key). 

We can now proceed to add in the data from the other three rows in our table: 

»> people[’Arthur'] = { 'Name': 'Arthur Dent', 

'Gender': 'Male', 

'Occupation': 'Sandwich-Maker' 

'Home Planet': 'Earth' } 

»> people ['Trillian' ] = { 'Name': 'Tricia McMillan', 

'Gender': 'Female', 

'Occupation': 'Mathematician 

'Home Planet': 'Earth' } 

»> people['Robot'] = { 'Name': 'Marvin', 

Gender': 'Unknown', 

Occupation': 'Paranoid Android 

Planet': 'Unknown' } 

A/jav-vm's data * associated W.th 
the "Robot" key- 




Tt-idias data 
is assodiated 

'with the 
"Trillian" key. 



you are here ► 
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it’s just data 


A Pictiowary of Pictionaries (a.k.a. a Table) 

With the people dictionary populated with four embedded dictionaries, we can 
ask the interpreter to display the people dictionary at the >>> prompt. 

Doing so results in an unholy mess of data on screen (see below). 

Despite the mess, all of our data is there. Note that each opening curly brace starts 
a new dictionary, while a closing curly brace terminates a dictionary. Go ahead and 
count them (there are five of each): 

»> people 

{'Ford 1 : {'Occupation': 'Researcher', 'Gender': 

'Home Planet': 'Betelgeuse Seven', 'Name': 'Ford Prefect'}, 
'Trillian': {'Occupation': 'Mathematician', 'Gender': 
'Female', 'Home Planet': 'Earth', 'Name': 'Tricia 
McMillan'}, 'Robot': {'Occupation': 'Paranoid Android', 

'Gender': 'Unknown', 'Home Planet': 'Unknown', 'Name': 

'Marvin'}, 'Arthur': {'Occupation': 'Sandwich-Maker', 

'Gender': 'Male', 'Home Planet': 'Earth', 'Name': 'Arthur 
Dent'}} 


lt' s a Mile hard 
■k read, bui all 
daia is there. 




The interpreter just 
dumps the data to the screen. 
Any chance we can make this 
more presentable? 


Yes, we can make this easier to read. 

We could pop over to the >>> prompt and code 
up a quick for loop that could iterate over each 
of the keys in the people dictionary. As we did 
this, a nested for loop could process each of 
the embedded dictionaries, being sure to output 
something easier to read on screen. 

We could...but we aren’t going to, as someone else 
has already done this work for us. 


138 Chapter 3 





structured data 


Pretty-Prmtmg Complex Pata Structures 


The Standard library includes a module called pprint that can take any data 
structure and display it in a easier-to-read format. The name pprint is a 
shorthand for “pretty print.” 

Let’s use the pprint module with our people dictionary (of dictionaries). 
Below, we once more display the data “in the raw” at the >>> prompt, and then 
we import the pprint module before invoking its pprint function to produce 
the output we need: 


didtionav-y 
is hard io v-fad- 


»> people 

{'Ford': {'Occupation': 'Researcher', 'Gender': 'Male', 

'Home Planet': 'Betelgeuse Seven', 'Name': 'Ford Prefect'}, 
'Trillian': {'Occupation': 'Mathematician', 'Gender': 
'Female', 'Home Planet': 'Earth', 'Name': 'Tricia 
McMillan'}, 'Robot': {'Occupation': 'Paranoid Android', 

'Gender': 'Unknown', 'Home Planet': 'Unknown', 'Name': 
'Marvin'}, 'Arthur': {'Occupation': 'Sandwich-Maker', 

'Gender': 'Male', 'Home Planet': 'Earth', 'Name': 'Arthur 
Dent'}} 

»> 

»> import pprint -- l»>port the "pprint" module, then invoke 

the "pprint” -Pundtion to do the work. 



»> 


»> pprint .pprint (people) 


Ford 


{'Arthur': {'Gender': 'Male', 

'Home Planet': 'Earth', 

'Name': 'Arthur Dent', 

'Occupation': 'Sandwich-Maker'}, 

{'Gender': 'Male', 

'Home Planet': 'Betelgeuse Seven', 

'Name': 'Ford Prefect', 

'Occupation': 'Researcher'}, 

{'Gender': 'Unknown', 

'Home Planet': 'Unknown', 

'Name': 'Marvin', 

'Occupation': 'Paranoid Android'}, 

'Trillian': {'Gender': 'Female', 

'Home Planet': 'Earth', 

'Name': 'Tricia McMillan', 

'Occupation': 'Mathematician'}} 


Robot 


This output 
is mudh easifc 
°n the eye- 
N°lc that we 
stili have -five 

- opehing and tive 

dlosing durly 

braces. |ts just 
that—thanks to 
"pprint”—they 

are how SO 

edsiev* -to see 
£ouh-t). 


you are here ► 
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how it looks 


Visualizing Complex Rata Structores 

Let’s update our diagram depicting what the interpreter now “sees” when the 
pe opi e dictionary of dictionaries is populated with data: 

The V 0 ?'*” 

dittiona vy 


/Wtbuv 


Ford 


people 


Robot 


Tvillia* 



£a*dv/iek 


— Alakev-^ 


Male 


Barii) 


£ /Wtbuv- Dent 




cuse Severi 


| Ford Pre-Pedt 



{pav-anoid /Wvoid^ 


Wnknovm 


Wnknown 


Mavvm 



| Matbematidian ^ 


Ferale 


Bari\\ 


hrMa MdMillan 


At this point, a reasonable question to ask is: Now that we have ali this data stored in a 
dictionary of dictionaries , how do we get at it? Let’s answer this question on the next page. 





embedded 
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structured data 


Accessing a Complex Pata Structure s Pata 

We now have our table of data stored in the pe opi e dictionary. Let’s remind 
ourselves of what the original table of data looked like: 


Name 

Gender 

Occupation 

Home Planet 

Fo\rd Pve-fedi 

Male 

Reseav-dhev- 


Avtkuv Derrt 

Male 

Sandwidh-Makev- 

Eav-tK 

Tv-idia MdMilla* 

Ferale 

Mathcr^atidia^ 

Eav-tK 

Mav-vm 

Uhkhowh 

Pava^oid Andvoid 

Mnknown 


If we were asked to work out what Arthur does, we’d start by looking down the 
Name column for Arthur’s name, and then we’d look across the row of data until 
we arrived at the Occupation column, where we’d be able to read “Sandwich- 
Maker.” 


When it comes to accessing data in a complex data structure (such as our pe opi e 
dictionary of dictionaries), we can follow a similar process, which we’re now going 
to demonstrate at the >>> prompt. 

We start by finding Arthur’s data in the pe opi e dictionary, which we can do by 
putting Arthur 5 s key between square brackets: 


-V 

»> people [' Arthur f ] 

{'Occupation 1 : 'Sandwich-Maker 1 , ' Home 

'Gender 1 : 'Male 1 , 1 Name 1 : 'Arthur Dent' 

Having found Arthur’s row of data, we can now ask for the value associated with 
the Occupation key To do this, we employ a second pair of square brackets to 
index into Arthur’s dictionary and access the data we’re looking for: 


/\sk -Cor 

ArtWrs 

vov/ 

da^^a* 



Planet f : f Earth ! , ^ 
} 

"The irow ot didtiohavy - 
da-fca assodia-fced wi-fch -the 

'A*"thwr H key 


Identity the vox- 




Idenl ity -the dolur»*. 


»> people['Arthur']['Occupation'] 
'Sandwich-Maker' 


Using double square brackets lets you access any data value from a table by 
identifying the row and column you are interested in. The row corresponds to a 
key used by the enclosing dictionary (people, in our example), while the column 
corresponds to any of the keys used by an embedded dictionary 


you are here ► 
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complex wrap-up 


Pata Is As Complex As You Make It 

Whether you have a small amount of data (a simple list) or something more 
complex (a dictionary of dictionaries), it 5 s nice to know that Pythoifs four 
built-in data structures can accommodate your data needs. What 5 s especially 
nice is the dynamic nature of the data structures you build; other than tuples, 
each of the data structures can grow and shrink as needed, with Python 5 s 
interpreter taking care of any memory allocation/deallocation details for you. 

We are not done with data yet, and we’11 come back to this topic again later in 
this book. For now, though, you know enough to be getting on with things. 

In the next chapter, we start to talk about techniques to effectively reuse code 
with Python, by learning about the most basic of the code reuse technologies: 
functions. 
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Chapter Code, 1 of 2 


vowels = ['a', 'e', 'i', 'o', 'u'] 

word = input("Provide a word to search for vowels: ") 

found = {} 

found[ ' a ' ] =0 
found['e'] =0 
found[ ' i ' ] =0 
found['o'] =0 
found [ ' u ' ] =0 

for letter in word: 

if letter in vowels: 

found[letter] += 1 

for k, v in sorted(found.items()): 

print(k, 'was found’, v, 'time(s).') 


Tor 

Wls^py", w hich 
pen-Pov-med a (<rc<\{*er\ty 
douivt. This iode was 
Ooosely) based oh 
"vowels?.py", w hidh we 
saw ih Chap-tev 1. 


|n an att«»ft ■fco 
remove the ditWavy 
'mi-tialization tode, we 
treated "vowels^.py , 

wbitb dvashed wi-tv> a— 

runtime evvov (due -to 
ws -failin^ "to mvtiali^e 

•bbe -fve^venty ^ ow, '"t s ’ 


vowels = [’a’, ’e’, ’i’, 'o', ’u’] 

word = input("Provide a word to search for vowels: ") 

found = {} 

for letter in word: 

if letter in vowels: 

found[letter] += 1 

for k, v in sorted(found.items()): 

print(k, 'was found', v, 'time(s).') 


vowels = ['a', 'e', 'i', 'o', 'u'] 

word = input("Provide a word to search for vowels: ") 

found = {} 

for letter in word: 

if letter in vowels: 

found.setdefault(letter, 0) 
found[letter] += 1 

for k, v in sorted(found.items()): 

print(k, 'was found', v, 'time(s).') 


u vowelsk^y 

CV-VOV- -tha^ks -to ^ 

use of ‘the 'seWc-Cault 
mctkod, wkitk tomes Wrtk 
' every dittio^avy (a*d assio^s 
a de-fault value a key »•£ a 
value is* ■t alv-eady set). 
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the code 


Chapter Code, 2 of 2 


vowels = set('aeiou') 

word = input("Provide a word to search for vowels: ") 
found = vowels.intersection(set(word)) 
for vowel in found: 
print(vowel) 



-Pinal vevsion ot -the vowels 

P' ro 9' r *"» WlsTpy", -Uk 

P ^ hoh ' s *t data 
t uefaw fc eowidevably shv-i»k 
the list-based 'WlsS.py" tode 

™*thohaliiy. 


O 


Was there no sample 
program that took 
advantage of tuples? 



No, there wasn’t. But that’s OK. 

We didn’t exploit tuples in this chapter with an 
example program, as tuples don’t come into their 
own until discussed in relation to functions. As we 
have already stated, we’11 see tuples again when 
we meet functions (in the next chapter), as well 
as elsewhere in this book. Each time we see them, 
we’ll be sure to point out each tuple usage. As 
you continue with your Python travels, you’11 see 
tuples pop up all over the place. 


t 
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4 co c [e retise 




Functions and Modules 




No matter how much code I 
wri+e, things just become totally 
unmanageable after a while... 








Reusing code is key to building a maintainable system. 

And when it comes to reusing code in Python, it all starts and ends with the humble 
function. Take some lines of code, give them a name, and youVe got a function (which 
can be reused). Take a collection of functions and package them as a file, and youVe 
got a module (which can also be reused). It’s true what they say: it’s good to share, and 
by the end of this chapter, you’ll be well on your way to sharing and reusing your code, 
thanks to an understanding of how Python’s functions and modules work. 
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starting with functions 


Reusing Code with Functions 


Although a few lines of code can accomplish a lot in Python, sooner or later 
you’re going to find your program’s codebase is growing...and, when it does, 
things quickly become harder to manage. What started out as 20 lines of 
Python code has somehow ballooned to 500 lines or more! When this happens, 
it’s time to start thinking about what strategies you can use to reduce the 
complexity of your codebase. 


Like many other programming languages, Python supports modularity, in 
that you can break large chunks of code into smaller, more manageable pieces. 
You do this by creating functions, which you can think of as named chunks 
of code. Recall this diagram from Ghapter 1, which shows the relationship 
between functions, modules, and the Standard library: 



...is pav“t o-C a 



...whith tomes as \>art <£ 
-the standardi Wwavy 



ln ihis f ha pkv, we ave CmttKbrahM 
dveatmg and usi.g Wtio.s (but 

±1 dia 9 ' raw ' 

Chaptev / ■» -this ms-ta,e e to « mihd you 

how WWs U the lavgev sehLe 

f u x 9 ^ wil1 ouv ow„ module 

too, but ave leavi.0 the dveatio. ot 
libvavies to othev books. 



getcwd chmod 


enum 


In this chapter, we’re going to concentrate on what’s involved in creating your 
own functions, shown at the very top of the diagram. Once you’re happily 
creating functions, we’ll also show you how to create a module. 
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code reuse 


Iwtroducmg Functions 


Before we get to turning some of our existing code into a function, let 5 s spend a 
moment looking at the anatomy of any function in Python. Once this introduction is 
complete, wedl look at some of our existing code and go through the steps required to 
turn it into a function that you can reuse. 

Don’t sweat the details just yet. All you need to do here is get a feel for what functions 
look like in Python, as described on this and the next page. We’ll delve into the details 
of all you need to know as this chapter progresses. The IDLE window on this page 
presents a template you can use when creating any function. As you are looking at it, 
consider the following: 

(3 Functions introduce two new keywords: def and return 

Both of these keywords are colored orange in IDLE. The def keyword names the function 
(shown in blue), and details any arguments the function may have. The use of the return 
keyword is optional, and is used to pass back a value to the code that invoked the function. 

o Functions can accept argument data 

A function can accept argument data (i.e., input to the function). You can specify a list of 
arguments between the parentheses on the de f line, following the function’s name. 

O Functions contain code and (usually) documentation 

Code is indented one level beneath the def line, and should include comments where it 
makes sense. We demonstrate two ways to add comments to code: using a triple-quoted 
string (shown in green in the template and known as a docstring), and using a single-line 
comment, which is prefixed by the # Symbol (and shown in red, below). 


The MeJ* 

“the *fun£-tion 3 nd lists 
a»y av^ur*Cirrts- 


function.template.py - /Users/Paul/Desktop/NewBook/ch04/function template.py (3.4.3) 


The do£s-fcv"ihg” 
desoibes -the 

-Puh£-fci 


on s purpose 



def a_descri.pt ive_name (optional_arguments) : 
"""A documentation string.""" 

# Your function's code goes here. 

# Your function's code goes here. 


# Your function's code goes here. 
return optional_value 


Ln: 8 Coi: 0 

■fir 




h handy 
‘templa-te 



ybur dode yes 

hev-e (i* pla^e 
-these sm^le- 
Ime tommen-t 

plaeeholdevs). 


Python uses the name "function" to describe a reusable chunk of code. Other programming languages 
use names such as"procedure/ , "subroutine,"and "method."When a function is part of a Python class, 
ifs known as a "method." You'll learn all about Python's classes and methods in a later chapter. 
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what about type? 


What About Type Information? 

Take another look at our function template. Other than some code to execute, 
do you think there 5 s anything missing? Is there anything you’d expect to be 
specified, but isn’t? Take another look: 


# # # function_template.py - /Users/Paul/Desktop/_NewBook/ch04/function_template.py (3.4.3) 

def a descriptive name (optional arguments): 

" " "A documentation string. " " 11 

# Your function's code goes here. 

# Your function's code goes here. 

# Your function's code goes here. 
return optional_value 



Ln: 8 Coi: 0 




|s thcvc 

missing pror» 
this 'Puhdiioh 


template? 


0 


I'm a little freaked out by 
that function template. How does 
the interpreter know what types the 
arguments are, as well as what type the 
return value is? 



It doesn’t know, but don’t let that worry you. 

The Python interpreter does not force you to specify the 
type of your function’s arguments or the return value. 
Depending on the programming languages you Ve used 
before, this may well freak you out. Don’t let it. 

Python lets you send any object as a argument, and pass 
back any object as a return value. The interpreter doesn’t 
care or check what type these objects are (only that they are 
provided). 

With Python 3, it is possible to indicate the expected types for 
arguments/return values, and we’ll do just that later in this 
chapter. However, indicating the types expected does not 
‘magically” switch on type checking, as Python never checks 
the types of the arguments or any return values. 
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code reuse 


Nammg a Chunk of Code with "def" 


Once youVe identified a chunk of your Python code you want to reuse, it 5 s 
time to create a function. You create a function using the def keyword 
(which is short for dejine). The def keyword is followed by the functioni 
name, an optionally empty list of arguments (enclosed in parentheses), a 
colon, and then one or more lines of indented code. 


Recall the vowels7 . py program from the end of the last chapter, which, 
given a word, prints the vowels contained in that word: 


Take a set ot 
vowels— 

..a»d a wovdl - '” 


•••■fcheb pe\r-Po\rr* 
ih-tarsee-tioh. 


vowels = set('aeiou') 

word = input("Provide a word to search for vowels 
found = vowels.intersection(set(word) ) 
for vowel in found. . 

pisplay a*y 
v-esufc 


") 


print(vowel) 


1 



Let 5 s imagine you plan to use these five lines of code many times in a much 
larger program. The last thing you’11 want to do is copy and paste this code 
everywhere it’s needed...so, to keep things manageable and to ensure you 
only need to maintain one copy of this code, let 5 s create a function. 

We’ll demonstrate how at the Python Shell (for now). To turn the above five 
lines of code into a function, use the de f keyword to indicate that a function 
is starting; give the function a descriptive name (always a good idea); provide 
an optionally empty list of arguments in parentheses, followed by a colon; 
and then indent the lines of code relative to the def keyword, as follows: 


Take tke time 
to ckoose a gooct 
descriptive name 
lor your function. 


Stari Witk -tke 
"det” ke-ywovd 


6)ive VouV* -funfrtion 3 nite» 
destvipwe na»>>e 





»> def search4vowels () 



FV~de a» optio*,! | lst 0 f tkis ta s , it 

Wti.» kas argumcts, 


Don'-t -fovge-t 
tke dolon. 


f ~vowels = set('aeiou') 

word = input("Provide a word to search for vowels: 


Tke -fives lines 
ot dode Won> 
tke 'Wlsl-r/' 
Dvoyan») s«*i"taV>ly 
indented 


1 found = vowels.intersection(set(word)) 
for vowel in found: 
print(vowel) 





TiA/ICt to dontivn» tkat tke mdentefl c.oae 



Now that the function exists, let’s invoke it to see if it is working the way we 
expect it to. 
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calling functions 


Invoking Your Function 

To invoke functions in Python, provide the function name together with 
values for any arguments the function expects. As the search4vowels 
function (currently) takes no arguments, we can invoke it with an empty 
argument list, like so: 

»> search4vowels () 

Provide a word to search for vowels: hitch-hiker 

e 

■ 

i 

Invoking the function again runs it again: 

»> search4vowels () 

Provide a word to search for vowels: galaxy 
a 

There are no surprises here: invoking the function executes its code. 


Edit your function in an editor not at the prompt 

At the moment, the code for the search4vowels function has been 
entered into the >>> prompt, and it looks like this: 


»> 

0uv 'fiAKfction J 

as errtevedi ^ 
at the shell 
pvomnft- 


def search4vowels(): 

vowels = set( f aeiou f ) 

word = input( 11 Provide a word to search for vowels: 
found = vowels.intersection(set(word)) 
for vowel in found: 
print(vowel) 


In order to work further with this code, you can recall it at the >>> prompt 
and edit it, but this becomes very unwieldy, very quickly. Recall that once the 
code you’re working with at the >>> prompt is more than a few lines long, 
you’re better off copying the code into an IDLE edit window. You can edit it 
much more easily there. So, let’s do that before continuing. 

Create a new, empty IDLE edit window, then copy the function’s code from 
the >>> prompt (being sure not to copy the >>> characters), and paste it into 
the edit window. Once you’re satisfied that the formatting and indentation are 
correct, save your file asvsearch.py before continuing 


Be sure you Ve 
saved your code 
as "vsearch.py” 
alter copying tl 
functions code 
from the shell. 
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Use U?LE'$ Editor to Make Changes 

Here’s what the vsearch. py file looks like in IDLE: 



The -PuKd-tiohs tode is 

ho ' w i» ah IDLE edi-t 

wihdow, ahd has been 
saved as Veav-th.py". 


# ' ■ O vsearch,py - /Users/Paul/Desktop/NewBook/ch04/vseafch.py (3.4,3) 

def search4vowels (): 

vowels = set( aeiou 1 ) 

word = input ( 11 Provide a word to search for vowels: 
found = vowels-intersection (set (word)) 
for vowel in found: 
print (vowel) 




Ln:fl 

Coi: 0 



If you press F5 while in the edit window, two things happen: the IDLE shell 
is brought to the foreground, and the shell restarts. However, nothing appears 
on screen. Try this now to see what we mean: press F5. 

The reason for nothing displaying is that you have yet to invoke the function. 
We’ll invoke it in a little bit, but for now let 5 s make one change to our function 
before moving on. It’s a small change, but an important one nonetheless. 

Let’s add some documentation to the top of our function. 


^ |$ iDLfc displays a* 

Yihen you pve» fV doh t 
oar.it 1 ReWh to edit 
Wmdovj ahd thetk that yo«r 
tode is the e*att sa««e as 
ouvs, theh tvy a^aih. 


To add a multiline comment (a docstring) to any code, enclose your 
comment text in triple quotes. 

Here’s the vsearch.py file once more, with a docstring added to the top of 
the function. Go ahead and make this change to your code, too: 


f\ dofcstvmj has been ^ 
added bo ‘the s 

tode, xhith (bvietly) 
destribes the pwvpose 

ot this -fiAhttioh- 


# # O vsearch, py - / i Users/Paul/Desktop/New6ook/ch04/vsea/ch.,py {3. 4. 3) 


def search4vowels (): 

1 .Display any vowels found in an asked-for word. 111111 ^- 

vowels = set ( 1 aeiou f ) 

word = input ( "Provide a word to search for vowels: 11 ) 
found = vowels*intersection (set (word)) 
for vowel in found: 
print (vowel) 

H 


Ln: 9 Co 

:0 
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whither PEP compliance? 


What'$ thc Peal with AII Tbose Strings? 

Take another look at the function as it currently stands. Pay particular attention to 
the three strings in this code, which are all colored green by IDLE: 


vsearch.py - /Users/Paul/Desktop/ .NewBook/ch04/vsearch,,py (3.4.3) 


def search4vowels {): 

1111 "Display any vowels found in an asked-for word. 1,1111 
vowels = set { 1 aeiou 1 ) 
word = input ( 11 Provide a word to search for vowels: 11 ) 
found = vowels*intersection (set (word)) 
for vowel in found: 
p rint (vowe1) 



Understanding the string quote characters 


|pLfc's sbows ibat bave 

a dons.sWv problem W.ib ouv- use oX sirm^ 


-Lmr I.A/k#W An USC 


In Python, strings can be enclosed in a single quote character ('), a double quote 
character ("), or what’s known as triple quotes (" " " or 1 ’ 1 ). 

As mentioned earlier, triple quotes around strings are known as docstrings, 
because they are mainly used to document a function’s purpose (as shown above). 
Even though you can use " " " or 1 1 1 to surround your docstrings, most Python 
programmers prefer to use " " ". Docstrings have an interesting characteristic in 
that they can span multiple lines (other programming languages use the name 
“heredoc” for the same concept). 

Strings enclosed by a single quote character (') or a double quote character (") 
cannot span multiple lines: you must terminate the string with a matching quote 
character on the same line (as Python uses the end of the line as a statement 
terminator). 

Which character you use to enclose your strings is up to you, although using the 
single quote character is very popular with the majority of Python programmers. 
That said, and above all else, your usage should be consistent. 

The code shown at the top of this page (despite being only a handful of lines of 
code) is not consistent in its use of string quote characters. Note that the code runs 
fine (as the interpreter doesn’t care which style you use), but mixing and matching 
styles can make the code harder to read than it needs to be (which is a shame). 


Be consistent in 
your use of string 
cruote characters. 
If possiUe, use 

e cruotes. 
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Follow Pest Praefice As Per the PEPs 


When it comes to formatting your code (not just strings), the Python programming 
community has spent a long time establishing and documenting best practice. This 
best practice is known as PEP 8. PEP is shorthand for “Python Enhancement 
Protocol.” 

There are a large number of PEP documents in existence, and they primarily detail 
proposed and implemented enhancements to the Python programming language, 
but can also document advice (on what to do and what not to do), as well as describe 
various Python processes. The details of the PEP documents can be very technical and 
(often) esoteric. Thus, the vast majority of Python programmers are aware of their 
existence but rarely interact with PEPs in detail. This is true of most PEPs except for 
PEP 8. 

PEP 8 is the style guide for Python code. It is recommended reading for all Python 
programmers, and it is the document that suggests the “be consistent’ 5 advice for string 
quotes described on the last page. Take the time to read PEP 8 at least once. Another 
document, PEP 257, offers conventions on how to format docstrings, and it’s worth 
reading, too. 


Finct tlie list 

ol PEPs here: 

https:/ / www. 
python.org/ 
dev/peps/. 


Here is the search4vowels function once more in its PEP 8— and PEP 257- 
compliant form. The changes arefft extensive, but standardizing on single quote 
characters around our strings (but not around our docstrings) does look a bit better: 


This is a PEP 

2-57-£or*p|ia h -t do^stv-ihj. 


vsearch.py - /Users/Paul/Desktop/_NewBook/ch04/vsearch.py (3.4.3) 


def search4vowels (): 

"""Display any vowels found in an asked-for word.""" 
vowels = set (' aeiou * ) 

word = input ( ’Provide a word to search for vowels: ) 

found = vowels.intersection (set (word)) 
for vowel in found: 
print (vowel) 


Of course, you don’t have to write code that conforms exactly to PEP 8. For example, 
our function name, search4vowels, does not conform to the guidelines, which 
suggests that words in a functioffs name should be separated by an underscore: 
a more compliant name is search for_vowels. Note that PEP 8 is a set of 
guidelines, not rules. You don’t have to comply, only consider, and we like the name 
search4vowels. 



W e* ve heeded PEP 
@*s adviee oy\ bemg 
eonsisEerrt v/rth “the sm^lc 
ehava^tev we use 
{jo suvroufid ouV" sfarinjs. 


That said, the vast majority of Python programmers will thank you for writing code 
that conforms to PEP 8, as it is often easier to read than code that doesfft. 

Let’s now return to enhancing the search4vowels function to accept arguments. 
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add an argument 


Functions Can Accept Arguments 

Rather than having the function prompt the user for a word to search, let’s change 
the search4vowels function so we can pass it the word as input to an argument. 

Adding an argument is straightforward: you simply insert the argument 5 s name 
between the parentheses on the de f line. This argument name then becomes a 
variable in the function’s suite. This is an easy edit. 

Let 5 s also remove the line of code that prompts the user to supply a word to search, 
which is another easy edit. 

Let 5 s remind ourselves of the current state of our code: 


RememLer' 

ff V • 

suite is 
Pytkon-speak 
lor "hlock.” 


vsearch „ py - /U se rs/Pa u l/Deskto p/N ewBo o k/c h 04/vsearc h, py (3,4,3) 




Heve s ouv 

ovi^mdt 

«functio*. 


def search4vowels (): 

.Display any vowels found in an asked-for word. 111111 

vowels = set ( 1 aeiou 1 ) 

word = input ( "Provide a word to search for vowels: 1 ) 
found = vowels.intersection (set (word)) 
for vowel in found: 
print (vowel) 



Applying the two suggested edits (from above) to our function results in the IDLE 
edit window looking like this (note: weVe updated our docstring, too, which is always 
a good idea): 


This lihc is^-fc 

^eeded 3hymov“c» 


Pu-t ‘the 




between “the 

pavevrtheses. 


'fc^ftarch.py - /Users/Paul/Desktop/ NewBook/ch04/vsearch.py (3,4,3) 



def search4vowels (word): 

1111 "Display any vowels found in a supplied word. 111111 
vowels = set ( 1 aeiou 1 ) ^ ^ The eall -to -the 

found = vowels . intersection (set (word))^ ^-*Wu-t w -Puheiioh is 

for vowel in found: goh C (as we do*-t 

print (vowel) heed ^ (jhe j 

todc dhymoire)- 



Be sure to save your file after each code change, before pressing F5 to take the new 
ver sion of your function for a spin. 
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Tqst DriVq 


your code loaded into IDLE’s edit window (and saved), press F5, then invoke the function a few 
times and see what happens: 




vsearch.py - AJser^Paul/Desktop/Ne wBook/eh04/vsearc h. py [3.4.3J 


The fcuvuwl 
dodc 


L 


* * 


def search4vowels (word): 

1MM, Display any vowels found in a supplied word. 111111 
vowels = set ( 1 aeiou 1 ) 

found = vowels.intersection (set (word)) 
for vowel in found: 
print (vowel) 

Python 3.4.3 Shell 


»> ================================ RESTART =========================== 

»> 

»> search4vowels() 

Iraceback (most recent call last): 

File "<pyshell#3>", line 1, in <module> 
search4vowels() 

TypeError: search4vowels() missing 1 required positional argument: 'word 

>» search4vowels ( 1 hitch-hiker ’ ) 

e 
■ 

i 

»> search4vowels(' hitch-hiker' , 'galaxy’ ) 

Traceback (most recent call last): 

File "<pyshell#5>", line 1, in <module> 
search4vowels( 1 hitch-hiker', 'galaxy') 

TypeError: search4vowels() takes 1 positional argument but 2 were given 

»> I 


Ln: 

112 

Coi: 4 





b, tt. t, iL why «d, .f w ^ 

tliereiAre no 

- Durnb Questipris - 



Am I restricted to only a single argument when creating functions in Python? 


No, you can have as many arguments as you want, depending on the Service your function is providing. We are deliberately starting 
off with a straightforward example, and we’ll get to more involved examples as this chapter progresses. You can do a lot with arguments to 
functions in Python, and we plan to discuss most of what’s possible over the next dozen pages or so. 
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return a value 


FuwctioHS Return a Resuit 


As well as using a function to abstract some code and give it a name, 
programmers typically want functions to return some calculated value, which 
the code that called the function can then work with. To support returning a 
value (or values) from a function, Python provides the return statement. 

When the interpreter encounters a return statement in your functioni suite, 
two things happen: the function terminates at the return statement, and 
any value provided to the return statement is passed back to your calling 
code. This behavior mimics how return works in the majority of other 
programming languages. 

Let 5 s start with a straightforward example of returning a single value from 
our search4vowels function. Specifically, let’s return either True or 
False depending on whether the word supplied as an argument contains 
any vowels. 

This is a bit of a departure from our functioni existing functionality, but bear 
with us, as we are going to build up to something more complex (and useful) 
in a bit. Starting with a simple example ensures we have the basies in place 
first, before moving on. 
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The truth is... 

Python comes with a built-in function called bool 
that, when provided with any value, telis you whether 
the value evaluates to True or False. 

Not only does bool work with any value, it works 
with any Python object. The effect of this is that 
Python’s notion of truth extends far beyond the 1 for 
True and the 0 for False that other programming 
languages employ. 

Let’s pause and take a brief look at True and False 
before getting back to our discussion of return. 













code reuse 


- Truth TJp Cl«se 

Every object in Python has a truth value associated with it, in that the object 
evaluates to either True or False. 



Something is False if it evaluates to 0, the value None, an empty string, 
or an empty built-in data structure. This means all of these examples are 
False: 


»> bool (0) 

False C+ 

»> bool (0.0) 
False 

»> bool (' ') ^ 

False / 

»> bool ([ ]) 

False \ 

»> bool ({}) J 

False 

»> bool (None) ^ 
False 


/-P ah objed-t evaluates -fco 
'*t is always False. 


A* e^pty s-brih<), ah emp-ty lis-t, and 
ah empty didtiohavy all evaluate io 
False. 


PytWs u Kcmc W value is 
aUays False. 


Every other object in Python evaluates to True. Here are some examples of 
objects that are True: 


»> bool (1) ^ 

True 

»> bool (-1) 
True 

»> bool (42) 
True 


A hurwbeir that ish't O is always 
Ttue, eveh whe„ it's hegative. 


»> bool (0.0000000000000000000000000000001)^- 


A hoherwpty s-tv-ihg is 
always T\rue. 

J\ nonew>\>ty Wilt—•*» data 
strwdWe is True- 


True 

»> bool ( 1 Panic 1 ) 

True 

»> bool ([42, 43, 44]) 

True 

>»bool({'a': 42, 'b':42}) 

True 

We can pass any object to the bool function and determine whether it is 
True or False. 

Critically, any nonempty data structure evaluates to True. 


It may be 

veally small) 
but it > s 

st*U hO*t 0, 

so it*s TVuC- 
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handling the truth 


Returwmg One Value 

Take another look at our function’s code, which currently accepts any value 
as an argument, searches the supplied value for vowels, and then displays the 
found vowels on screen: 


def search4vowels(word): 

"""Display any vowels found in a supplied word.""" 
vowels = set('aeiou') 

found = vowels.intersection(set(word) ) 
for vowel in found 


print(vowel) 


Well these Uo lines. 


Ghanging this function to return either True or False, based on whether 
any vowels were found, is straightforward. Simply replace the last two lines of 
code (the for loop) with this line of code: 


return bool(found) 

Call the u bool w 
-fufiCtiott, a*d... 



•••pass ih the hdme ot 

the data stv-udtuv-e that 
eohtaihS the vesults o-p 
the vowels seareh. 


If nothing is found, the function returns False; otherwise, it returns True. 
With this change made, you can now test this new version of your function at 
the Python Shell and see what happens: 


The \eW* * 
state^ent (thanks 
to u booO 3 'wes vs 
eithev- u Tvue or 
“False”. 


»> search4vowels('hitch-hiker') 
True 

»> search4vowels ( 1 galaxy 1 ) 

True 

»> search4vowels (' sky') 

False 


As ih earlier 
^haptevs, we are 
ebssihg 
a vowel. 


If you continue to see the previous version’s behavior, ensure you Ve saved the 
new version of your function, as well as pressed F5 from the edit window. 




DorVt be tempted to put parentheses around the object that return passes back to the calling 
code. You don't need to. The return statement is not a function call, so the use of parentheses isn't a 
syntactical requirement. You can use them (if you reolly want to), but most Python programmers don't. 
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Returning More Than One Valoe 


Functions are designed to return a single value, but it is sometimes necessary 
to return more than one value. The only way to do this is to package the 
multiple values in a single data structure, then return that. Thus, you’re stili 
returning one thing, even though it potentially contains many individual 
pieces of data. 

Here’s our current function, which returns a boolean value (i.e., one thing): 


Noie: we ' ve 
upda-ted the 

don» ruent 


def search4vowels(word): r 

"""Return a boolean based on any vowels found.""" 
vowels = set( 1 aeiou') 

found = vowels.intersection(set(word)) 
return bool(found) 


It s a trivial edit to have the function return multiple values (in one set) as 
opposed to a boolean. Ali we need to do is drop the call to bool: 


def search4vowels(word): 

"""Return any vowels found in a supplied word.""" 
vowels = set('aeiou') 

found = vowels.intersection(set(word)) 

return found ^ 

~ - RtW the vesults as 3 

data stvuttuve (a set 


We ve upda-fced 
the eommeht 
ajain. 


We can further reduce the last two lines of code in the above version of our 
function to one line by removing the unnecessary use of the found variable. 

Rather than assigning the results of the intersection to the found 
variable and returning that, just return the intersection: 

def search4vowels(word): 

"""Return any vowels found in a supplied word.""" 
vowels = set('aeiou') 

return vowels . intersection (set (word)) < -n 

ReW> the data v/ithout the wse 
o£ -the uwetessavy "towr>d" variable- 


Our function now returns a set of vowels found in a word, which is exactly 
what we set out to do. 

However, when we tested it, one of our results has us scratching our head... 
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set weirdness 



Let’s take this latest version of the search4vowels function for a spin and see how it behaves. 
With the latest code loaded into an IDLE edit window, press F5 to import the function into the Python 
Shell, and then invoke the function a few times: 


# # + Python 3.4,3 Shell 

»> RESTART 

»> 

»> search4vowels ( 1 hitch-hiker 1 ) 

{'e', 'i'} 1 

»> search4vowels ( 1 galaxy 1 ) 

{'a'} 1 

»> search4vowels( 1 life r the universe and everything 1 ) i 

{'e', 'u\ 'a\ 'i'} 

»> search4vowels ( 1 sky r ) 

set() „/ 

»> 

| UtV o( these 

Tun£-tioh ihvodations 

f< -'«oirks as expetted, 

eveh though the vesult 
-rvorv» the last one 
looks a little weivd- 


Ln: 3S Coi: 4 


What'$ the deal with "setO"? 


Each example in the above Test Drive works fine, in that the function takes a 
single string value as an argument, then returns the set of vowels found. The 
one resuit, the set, contains many values. However, the last response looks a 
little weird, doesn’t it? Let’s have a closer look: 



i Oh 


You may have expected the function to retura { } to represent an empty set, 
but that 5 s a common misunderstanding, as { } represents an empty dictionary, 
not an empty set. 

An empty set is represented as set () by the interpreter. 

This may well look a little weird, but it’s just the way things work in Python. 
Let’s take a moment to recall the four built-in data structures, with a eye to 
seeing how each empty data structure is represented by the interpreter. 
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code reuse 


Recalling the Puilt-in Pata Structores 


Let’s remind ourselves of the four built-in data structures available to us. We’ll take 
each data structure in turn, working through list, dictionary, set, and finally tuple. 

Working at the shell, let 5 s create an empty data structure using the data structure built- 
in functions (BIFs for short), then assign a small amount of data to each. We’ll then 
display the contents of each data structure after each assignment: 



empty 

list 



»> 

»> 

[] 

»> 
»> 
[1 , 



M* the “list” BIF to 
detine a* empty list, 
then assijjh some data. 


BIF is short- 

kand lor "huilt- 
in lunction.” 


An empty ■ 
dictionavy 



Use -the "diei" BIF to 

detine a« empty dictionary, 
Ihen assign some data 



»> d = { 'first' : 1, 'second' : 2, 'third' : 
>» d 


{'second': 2, 'third': 3, 'first': 1} 


3 } 


An empty 
set 



»> s = set() 

>» s 
set () 

»> s = {1, 2, 3} fi. 
>» s 
{1, 2, 3} 



Kse the w set w B/F to 
de-fihe an empty set, 
then assign some data. 



Fveh thoujh sets ave endosed 
in £uvly bvades, so too ave 
dtetionavies. /U empty 
dtetionavy is alveady usinj 
the double £uvly bva£es, so 
a* empty set has to be 
vepvesented as w setO w . 


r 

An empty 
tuple 


»> t = tuple () — 
»> t _ 

^ 0 «=—— 

»> t = (1, 2, 3) <=: 

»> t 

d, 2, 3) 


: the “tuple” BIF to 
•ine an empty tuple, 
:n assign some data- 


Before moving on, take a moment to review 
how the interpreter represents each of the 
empty data structures as shown on this page. 


you are here ► 
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annotate your functions 


Use Annotations to Iwprove Your Pocs 


Our review of the four data structures confirms that the search4vowels function 
returns a set. But, other than calling the function and checking the return type, how 
can users of our function know this ahead of time? How do they know what to expect? 


A solution is to add this information to the docstring. This assumes that you very 
clearly indicate in your docstring what the arguments and return value are going 
to be and that this information is easy to find. Getting programmers to agree on a 
Standard for documenting functions is problematic (PEP 257 only suggests the format 
of docstrings), so Python 3 now supports a notation called annotations (also known 
as type hints). When used, annotations document—in a Standard way—the return type, 
as well as the types of any arguments. Keep these points in mind: 


o 

o 


Function annotations are optional 

It 5 s OK not to use them. In fact, a lot of existing Python code doesn’t (as they were only 
made available to programmers in the most recent versions of Python 3). 

Function annotations are informational 

They provide details about your function, but they do not imply any other behavior (such as 
type checking). 


Let’s annotate the search4vowels function’s arguments. The first annotation States 
that the function expects a string as the type of the word argument (:str), while the 
second annotation States that the function returns a set to its caller (-> set): 

VVe av-c statmj 'that the ^- 

"wovd” av^wmext is e*f>efrted--v y ** s ^' m 9 that the 

to be a stm 9 . \ Y «t»™ * *t to 

^ V-' its ealle*-. 

def search4vowels(word:str) -> set: 

MM "Return any vowels found in a supplied word. 11 n 11 
vowels = set( ! aeiou f ) 

return vowels.intersection(set(word)) 


Annotation syntax is straightforward. Each function argument has a colon appended 
to it, together with the type that is expected. In our example, : str specifles that the 
function expects a string. The return type is provided after the argument list, and is 
indicated by an arrow Symbol, which is itself followed by the return type, then the 
colon. Here -> set: indicates that the function is going to return a set. 

So far, so good. 

WeVe now annotated our function in a Standard way. Because of this, programmers 
using our function now know what’s expected of them, as well as what to expect from 
the function. However, the interpreter won ? t check that the function is always called 
with a string, nor will it check that the function always returns a set. Which begs a 
rather obvious question... 


For more details 
on annotations, 

see PEP 3107 

at lrttps://www. 
python.org/ dev/ 

peps/pep-3107/. 
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code reuse 


Why Use Function Annotations? 

If the Python interpreter isn’t going to use your annotations to check the types of 
your functioni arguments and its return type, why bother with annotations at all? 

The goal of annotations is not to make life easier for the interpreter; it’s to make 
life easier for the user of your function. Annotations are a documentation 
Standard, not a type enforcement mechanism. 

In fact, the interpreter does not care what type your arguments are, nor does it 
care what type of data your function returns. The interpreter calls your function 
with whatever arguments are provided to it (no matter their type), executes your 
functioni code, and then returns to the caller whatever value it is given by the 
return statement. The type of the data being passed back and forth is not 
considered by the interpreter. 

What annotations do for programmers using your function is rid them of the need 
to read your function’s code to learn what types are expected by, and returned 
from, your function. This is what theyTl have to do if annotations aren’t used. 

Even the most beautifully written docstring will stili have to be read if it doesn’t 
include annotations. 

Which leads to another question: how do we view the annotations without reading 
the functioni code? From IDLE 5 s editor, press F5, then use the help BIF at the 
>>> prompt. 


Use annotations 
to help document 
your lunctions, 
and use the "help" 

BIF to view them. 



If you haven’t done so already, use IDLE’s editor to annotate your copy of search4vowels, save 
your code, and then press the F5 key. The Python Shell will restart and the »> prompt will be waiting 
for you to do something. Ask the help BIF to display search4vowels documentation, like so: 


Python 3.4.3 Shell 


»> = 

»> 

»> help {search4vowels) 

Help on function search4vowels in module 


RESTART 


main 


search4vowels(word:str) -> set ^- 

Return any vowels found in a supplied word. 


Hoi only does "help" display 
the annotations, but it 
shows the do£string too. 



you are here ► 
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function recap 


Functions: What We Know Already 

Let’s pause for a moment and review what we know (so far) about Python functions. 



BULLET POINTS 


■ Functions are named chunks of 
code. 


Functions can accept any number of 
named arguments, including none. 


The def keyword is used to name 
a function, with the function’s code 
indented under (and relative to) the 
def keyword. 

Python’s triple-quoted strings can be 
used to add multiline comments to a 
function. When they are used in this 
way, they are known as docstrings. 


The return statement lets your 
functions return any number of 
values (including none). 

Function annotations can be used to 
document the type of your function’s 
arguments, as well as its return type. 


Let’s take a moment to once more review the code for the search4vowels function. 
Now that it accepts an argument and returns a set, it is more useful than the very first 
version of the function from the start of this chapter, as we can now use it in many 
more places: 


def search4vowels(word:str) -> set: 

1111 "Return any vowels found in a supplied word.""" 
vowels = set( ! aeiou f ) 

return vowels.intersection(set(word)) 



The mos-t 
rtU ht 
versioK o-f 
ou\r -fuhC-fcioh 


This function would be even more useful if, in addition to accepting an argument for 
the word to search, it also accepted a second argument detailing what to search for. 
This would allow us to look for any set of letters, not just the five vowels. 

Additionally, the use of the name word as an argument name is OK, but not great, 
as this function clearly accepts any string as an argument, as opposed to a single word. 
A better variable name might be phrase, as it more closely matehes what it is we 
expect to receive from the users of our function. 

Let’s change our function now to reflect this last suggestion. 
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code reuse 


Making a G-enerically Useful Function 

Here 5 s a version of the search4vowels function (as it appears in IDLE) after it 
has been changed to reflect the second of the two suggestions from the bottom of 
the last page. Namely, weVe changed the name of the word variable to the more 
appropriate phrase: 




vsearch.py - /Users/Paul/Desktop/ NewBook/ch04/vsearch.py (3,4,3) 


The u wo\rcT variable 
is how £alled u phrase 

\ 


def searchAvowels (phrase: str) -> set: 4 - 

,T11 "Ratum any vowels fotmd in a supplied phrase. 1,1111 
vowels = set ( 1 aeiou 1 ) 

return vowels.intersection (set (phrase)) 


The other suggestion from the bottom of the last page was to allow users to 
specify the set of letters to search for, as opposed to always using the five vowels. 
To do this we can add a second argument to the function that specifies the letters 
to search phrase for. This is an easy change to make. However, once we make 
it, the function (as it stands) will be incorrectly named, as we’ll no longer be 
searching for vowels, we’ll be searching for any set of letters. Rather than change 
the current function, let’s create a second one that is based on the first. Here’s 
what we propose to do: 




Sive the new function a more generic name 

Rather than continuing to adjust search4vowels, let’s create a new function called 
search41etters, which is a name that better reflects the new function’s purpose. 



Add a second argument 

Adding a second argument allows us to specify the set of letters to search the string for. Let’s 
call the second argument letters. And let’s not forget to annotate letters, too. 



Remove the vowels variable 

The use of the name vowels in the function’s suite no longer makes any sense, as we are 
now looking for a user-specified set of letters. 



Update the docstring 

There’s no point copying, then changing, the code if we don’t also adjust the docstring. Our 
documentation needs be updated to reflect what the new function does. 


We are going to work through these four tasks together. As each task is discussed, 
be sure to edit your vsearch.py file to reflect the presented changes. 


you are here ► 
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step by step 


Crcatiwg Awother Function, 1 of $ 

If you haven’t done so already, open the vsearch.py file in an IDLE edit window. 

Step 1 involves creating a new function, which we ? ll call search4 letters. Be 
aware that PEP 8 suggests that all top-level functions are surrounded by two blank 
lines. All of this book’s downloads conform to this guideline, but the code we show on 
the printed page doesn’t (as space is at a premium here). 

At the bottom of the file, type def followed by the name of your new function: 

# • ' ■ O vsearch.py -/Users/Paul/DeskitopdNewBook/chQ4/vsearch.py (3,4.3) 


def search4vowels (phrase: str) -> set: 

1111 "Return any vowels found in a supplied phrase.""" 
vowels = set ( 1 aeiou 1 } 

return vowels * intersection (set (phrase)) 


def search41etters 



For Step 2 we’re completing the function’s def line by adding in the names of the 
two required arguments, phrase and letter s. Remember to enclose the list of 
arguments within parentheses, and don’t forget to include the trailing colon (and the 
annotations): 


# O O vsearch.py VUsers/Paul/Deskitop/ NewBook/ch04/vsearch,py (3,4.3) 


def search4vowels (phrase: str) -> set: 

111111 Return any vowels found in a supplied phrase * 111111 
vowels = set ( 1 aeiou 1 ) 

return vowels * intersection( set (phrase)) 


def search41etters (phrase: str r letters:str) -> set: 



t_ Speditv the list of av$w»>ev>ts, a "d dont 

tLyt the tolon (and the annotations, too • 



'P LB ’ S ediU has ah t ici pated 

that the next Ime of Code needs to be indented 
(and automatidally positioned the du«oO? 


With Steps 1 and 2 complete, we’re now ready to write 
the function’s code. This code is going to be similar to 
that in the search4vowels function, except that we 
plan to remove our reliance on the vowels variable. 
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code reuse 


Creatiwg Awother Fuwctiow, 1 of $ 

On to Step 3, which is to write the code for the function in such a way as to 
remove the need for the vowels variable. We could continue to use the variable, 
but give it a new name (as vowels no longer represents what the variable does), 
but a temporary variable is not needed here, for much the same reason as why 
we no longer needed the f ound variable earlier. Take a look at the new line 
of code in search41etters, which does the same job as the two lines in 
search4vowels: 



# O 9 vsearch.py -/U5er3/Paul/De3ktop./NewBook/ch04/vsearch.py (3,4,3) 



def search4vowels (phrase: str) -> set: 

"""Ratum any vowels found in a supplied phrase. 111111 


Two Imcs 

r vowels = set ( 1 aeiou r ) 


o-f todt 
b tLor<\t ( 

-— return vowels.intersection (set (phrase)) 



def search41etters (phrase: str , letters: str) -> set: 
return set (letters).intersection (set (phrase)) 



Ln: 9 Coi: 0 



If that single line of code in search41etters has you scratching your head, 
don’t despair. It looks more complex than it is. Let’s go through this line of 
code in detail to work out exactly what it does. It starts when the value of the 
letters argument is turned into a set: 

set(letters) ^_ 


Oeate a set olojett 
W "I etters". 


This call to the set BIF creates a set object from the characters in the 
letters variable. We don’t need to assign this set object to a variable, as we 
are more interested in using the set of letters right away than in storing the set 
in a variable for later use. To use the just-created set object, append a dot, then 
specify the method you want to invoke, as even objects that aren’t assigned to 
variables have methods. As we know from using sets in the last chapter, the 
intersection method takes the set of characters contained in its argument 
phrase) and intersects them with an existing set object (letters): 


set(letters).intersection(set(phrase)) 



Pev-fov-r* a sei irrtev-sedtioin 
oy\ *tV>c sc*t objedt r»ade 
*fvor* "lettev-s* VrlV> 
sc*t objedt 'Pv-orw 
>bvasc . 


And, finally, the resuit of the intersection is returned to the calling code, thanks 
to the return statement: 


Ser.d tV>e resul-b 
katk to tte — 
tode- 


return set(letters).intersection(set(phrase)) 


you are here t 
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don’t forget! 


Crcatiwg Awother Function, $ of $ 

All that remains is Step 4, where we add a docstring to our newly created 
function. To do this, add a triple-quoted string right after your new functioni 
def line. Here’s what we used (as comments go it 5 s terse, but effective): 


ii 


/\ do&sbrm<) 



v&earch.py - /Users/Paul/Desktop/ NewBook/ch04/vsearch,py (3,4.3) 

def search4vowels (phrase: str) -> set: 

111111 Re tum any vowels found in a supplied phrase* 1111 
vowels = set {' aeiou 1 } 

return vowels *intersection (set (phrase)) 


def search41etters (phrase: str r letters: str) -> set: 

|' 11 "Return a set of the 'letters 1 found in 1 phrase 1 * 111111 
return set (letters)*intersection (set (phrase)) 



And with that, our four steps are complete and search41etters is 
ready to be tested. 


O 




Functions can hide complexity, too. 

It is correct to observe that weVe just created a one-line 
function, which may not feel like much of a “savings.” 
However, note that our function contains a complex single 
line of code, which we are hiding from the users of this 
function, and this can be a very worthwhile practice (not 
to mention, way better than all that copying and pasting). 

For instance, most programmers would be able to guess 
what search4letters does if they were to come 
across an invocation of it in a program. However, if 
they came across that complex single line of code in a 
program, they may well scratch their heads and wonder 
what it does. So, even though search4letters 
is “short,” it’s stili a good idea to abstract this type of 
complexity inside a function. 
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code reuse 



Save the vsearch.py file once more, and then press F5 to try out the search4letters function: 


Python 3,4,3 Shell 


RESTART 


»> = 

»> 

»> help (search41ett:ers} 

Help on function search.4 letters in module 


main 



Wsc the Vlp" BIF 

to led\ry\ how to use 

sea^eh^Hettevs^. 


search41etters(phrase:str, letters:str) -> set 

Return a set of the 1 letters 1 found in 1 phrase 1 . 

»> seareh4letters(' hitch-hiker 1 , 1 aeiou' ) 

{'e', -i-} 

»> search4letters ( 1 galaxy 1 , 1 xyz ) 

{ r x r , 'y'} 

»> search41etters( 1 life, the universe, and everything' , 
set () 

»> 


o f ) 


AII o-f these 
exa imples 
pv*odu£e what 
wc expeet them 

to. 



The search4letters function is now more generic than search4vowels, 
in that it takes any set of letters and searches a given phrase for them, rather 
than just searching for the letters a, e, i, o, and u. This makes our new 
function much more useful than search4vowels. Let’s now imagine that 
we have a large, existing codebase that has used search4vowels extensively. 
A decision has been made to retire search4vowels and replace it with 
search41etters, as the “powers that be” don’t see the need for both 
functions, now that search4letters can do what search4vowels does. 

A global search-and-replace of your codebase for the name “search4vowels” 
with ‘ C search41etters’ 5 won’t work here, as you’11 need to add in that second 
argument value, which is always going to be aeiou when simulating the 
behavior of search4vowels with searchUetters. So, for instance, this 
single-argument call: 

search4vowels("Don't panic!") 

now needs to be replaced with this dual-argument one (which is a much harder 
edit to automate): 

search41etters ( "Don f t panic!", 'aeiou’) 

It would be nice if we could somehow specify a default value for 
search41etters 5 s second argument, then have the function use it if no 
alternative value is provided. If we could arrange to set the default to aeiou, 
we’d then be able to apply a global search-and-replace (which is an easy edit). 
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revert automatically to 


Specifying Pefaolt Values for Arguments 

Any argument to a Python function can be assigned a default value, which can 
then be automatically used if the code calling the function fails to supply an 
alternate value. The mechanism for assigning a default value to an argument is 
straightforward: include the default value as an assignment in the functioni de f 
line. 

Here 5 s search41etters’s current def line: 


def search41etters(phrase:str, letters:str) -> set: 

This version of our functioni de f line (above) expects exactly two arguments, one 
for phrase and another for letters. However, if we assign a default value to 
letters, the function’s def line changes to look like this: 


def search41etters(phrase:str, letters:str= 1 aeiou 1 

We can continue to use the search41etters function in the same way as 
before: providing both arguments with values as needed. However, if we forget 
to supply the second argument (letters), the interpreter will substitute in the 
value aeiou on our behalf. 

If we were to make this change to our code in the vsearch . py file (and save it), 
we could then invoke our functions as follows: 



) -> set: 

de-fault value has heen^ 
assi ned bo "the letbers 
argumen*t and will he used 
whenever “the tallin^ to&t 
doesn t provide an aliev-na-te 

value- 


These “thy-ee 

-function talis 

all pvodute^? 
*the same 
v-esul*ts- 


»> 
{ 'a' 


»> 


{ 'a' 
»> 



search41etters( 1 life 
, 'e', 'i', 'u'} 

search41etters( 1 life 
, 'e', V} 

search4vowels( 1 life , 
, ’e', 1 i 1 , ’u’} 


the universe f and 
the universe r and 
the universe, and 


Not only do these function calls produce the same output, they also demonstrate 
that the search4vowels function is no longer needed now that the letters 
argument to search4letters supports a default value (compare the first and 
last invocations above). 


everything 1 ) 

everything 1 , 1 aeiou 


everything 1 ) 



In this invota-fcion, 
we are talling 
searth^vowels^ no-t 

searth^letfcers”. 


i 


Now, if we are asked to retire the search4vowels function and replace all 
invocations of it within our codebase with search41etters, our exploitation 
of the default value mechanism for function arguments lets us do so with a simple 
global search-and-replace. And we don’t have to use search41etters to only 
search for vowels. That second argument allows us to specify any set of characters 
to look for. As a consequence, search41etters is now more generic, and more 
useful. 
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code reuse 


Positional Versus Keyword Assignment 


As weVe just seen, the search41etters function can be invoked with 
either one or two arguments, the second argument being optional. If you 

provide only one argument, the letters argument defaults to a string of 0 u v -furfrtior s 

vowels. Take another look at the function’s def line: I mC -- 


def search41etters(phrase:str, letters:str= 1 aeiou 1 ) -> set: 


As well as supporting default arguments, the Python interpreter also lets 
you invoke a function using keyword arguments. To understand what a 
keyword argument is, consider how weVe invoked search41etters up 
until now, for example: 




def search41etters(phrase:str, letters:str= 1 aeiou 1 ) 


-> set: 


In the above invocation, the two strings are assigned to the phrase and 
letters arguments based on their position. That is, the first string is 
assigned to phrase, while the second is assigned to letters. This is known 
as positional assignment, as it’s based on the order of the arguments. 


In Python, it is also possible to refer to arguments by their argument name, 
and when you do, positional ordering no longer applies. This is known as 
keyword assignment. To use keywords, assign each string in any order to its 
correct argument name when invoking the function, as shown here: 




argumchts iimportant 
whch keyword arguments 
are used durmg ‘mvodaiioh. 


search4letters(letters= 1 xyz 1 , phrase= 1 galaxy') 




def search41etters(phrase:str, letters:str= 1 aeiou 1 ) -> set 


Both invocations of the search4 letters function on this page produce 
the same resuit: a set containing the letters y and z. Although it may be 
hard to appreciate the benefit of using keyword arguments with our small 
search41etters function, the flexibility this feature gives you becomes 
ciear when you invoke a function that accepts many arguments. Well see an 
example of one such function (provided by the Standard library) before the 
end of this chapter. 


you are here ► 
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a quick update 


Updating What We Kwow About Functions 

Let’s update what you know about functions now that youVe spent some time exploring 
how function arguments work: 



BULLET POINTS 


■ As well as supporting code reuse, 
functions can hide complexity. If you 
have a complex line of code you 
intend to use a lot, abstract it behind 
a simple function call. 


■ Any function argument can be 
assigned a default value in the 
function’s def line. When this 
happens, the specification of a value 
for that argument during a function’s 
invocation is optional. 


As well as assigning arguments by 
position, you can use keywords, 
too. When you do, any ordering is 
acceptable (as any possibility of 
ambiguity is removed by the use of 
keywords and position doesn’t matter 
anymore). 



These functions really 
hit the mark for me. 
How do I go about using 
and sharing them? 


y 


There’s more than one way to do it. 

Now that you have some code that’s worth 
sharing, it is reasonable to ask how best to use 
and share these functions. As with most things, 
there’s more than one answer to that question. 
However, on the next pages, you’11 learn how best 
to package and distribute your functions to ensure 
it’s easy for you and others to benefit from your 
work. 


A 
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code reuse 


Functions Peget Modules 

Having gone to all the trouble of creating a reusable function (or two, as is the 
case with the functions currently in our vsearch . py file), it is reasonable to ask: 
whaVs the best way to share functions? 

It is possible to share any function by copying and pasting it throughout your 
codebase where needed, but as that’s such a wasteful and bad idea, we arerft 
going to consider it for very much longer. Having multiple copies of the same 
function littering your codebase is a sure-fire recipe for disaster (should you ever 
decide to change how your function works). It 5 s much better to create a module 
that contains a single, canonical copy of any functions you want to share. Which 
raises another question: how are modules created in Python? 

The answer couldnd be simpler: a module is any file that contains functions. 
Happily, this means that vsearch . py is already a module. Here it is again, in all 
its module glory: 



module 


Share your 
Functions 
in modules. 


# O O vsearch,py - /Users/Paul/DesktopPNewBook/ch04/vsearch.py (3,4.3) 

def search4vowels (phrase: str) -> set: 

"""Ratum any vowels found in a supplied phrase * 111111 
vowels = set ( aeiou ) 

return vowels *intersection (set (phrase)) 

def search41etters (phrase: str , letters: str= 1 aeiou 1 ) -> set : 

1111 "Return a set of the 'letters 1 found in 'phrase 1 .""" 
return set (letters)*intersection (set (phrase)) 

H 


ILn: 10 

o 


'WtW ? v' *»*»'"* 

a -file, mato»* *t a Wly 



rr\ 


odulc- 


Creating modules couldvrt be easier, however... 


Creating modules is a piece of cake: simply create a file of the functions you want 
to share. 


Once your module exists, making its contents available to your programs is also 
straightforward: all you have to do is import the module using Python’s import 
statement. 


This in itself is not complex. However, the interpreter makes the assumption that 
the module in question is in the search path, and ensuring this is the case can 
be tricky. Let’s explore the ins and outs of module importation over the next few 
pages. 


you are here ► 
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where’s my module? 


How Are Modules Found? 

Recall from this book ? s first chapter how we imported and then used the 
randint function from the random module, which comes included as part 
of Python’s Standard library. Here 5 s what we did at the shell: 



module 


|derrb-fy “the 

{fi iw\^ov“b) 



»> import random 
»> random. randint (0, 255) 
42 R 


What happens during module importation is described in great detail in the 
Python documentation, which you are free to go and explore if the nitty- 
gritty details float your boat. However, all you really need to know are the 
three main locations the interpreter searches when looking for a module. 
These are: 


...'mvok t oy\C o$ 
"the module s 
-fund-bons. 



Your current working directory 

This is the folder that the interpreter thinks you are currently 
working in. 



Your interpreters site-packages locations 

These are the directories that contain any third-party Python 
modules you may have installed (including any written by you). 



The Standard library locations 

These are the directories that contains all the modules that make up 
the Standard library. 


The order in which locations 2 and 3 are searched by the interpreter can vary 
depending on many factors. But don’t worry: it is not important that you 
know how this searching mechanism works. What is important to understand 
is that the interpreter always searches your current working directory first , 
which is what can cause trouble when you’re working with your own custom 
modules. 



Depending on the operating 
system yotfre running, the 
name given to a location 
that holds files may be either 
directory or folder. We'll use 
"folder" in this book, except 
when we discuss the current 
working directory (which is a 
well-established term). 


To demonstrate what can go wrong, let 5 s run though a small exercise that is 
designed to highlight the issue. Here 5 s what you need to do before we begin: 


Create a folder called mymodules, which we’ll use to store your modules. It 
doesn’t matter where in your filesystem you create this folder; just make sure it 
is somewhere where you have read/write access. 

Move your vsearch.py file into your newly created mymodules folder. 
This file should be the only copy of the vsearch.py file on your computer. 


174 Chapter 4 















code reuse 


Ruwwmg Python f row the Command Une 

We’re going to run the Python interpreter from your operating systenTs 
command line (or terminal) to demonstrate what can go wrong here (even 
though the problem we are about to discuss also manifests in IDLE). 



If you are running any version of Windows , open up a command prompt and 
follow along with this session. If you are not on Windows , we discuss your 
platform halfway down the next page (but read on for now anyway). You 
can invoke the Python interpreter (outside of IDLE) by typing py -3 at the 
Windows C : \ > prompt. Note below how prior to invoking the interpreter, we 
use the cd command to make the mymodules folder our current working 
directory. Also, observe that we can exit the interpreter at any time by typing 
quit () at the >>> prompt: 




module 


^Chanje ih-fco the 
'mymodules" Qder. 


Py-tho* 'i- 




mpo\rt "the 
module--- 

Use ihe 
modulet —^ -• 
-Pun^ticms. 


File Edit Window Help Redmond #1 


C:\Users\Head First> cd mymodules 
C:\Users\Head First\mymodules> py -3 

Python 3.4.3 (v3.4.3:9b73fIc3e601, Feb 24 2015, 22:43:06) [MSC 
v.1600 32 bit (Intel)] on win32 

Type "help", "Copyright", "credits" or "license" for more Information 
»> import vsearch 

»> vsearch. search4vowels (’ hitch-hiker') 

{'i’, 'e'} 

»> vsearch. search41etters (' galaxy' , ' xyz ') 

{’Y\ 'x'} 

»> quit() 


£*rt tV>e Python 
mtev^v-etev and netuvn 
to youn o^evatin^ 
system’s donuwand fro-ft 


C:\Users\Head First\mymodules> 


1 




This works as expected: we successfully import the vsearch module, then 
use each of its functions by prefixing the function name with the name of 
its module and a dot. Note how the behavior of the >>> prompt at the 
command line is identical to the behavior within IDLE (the only difference is 
the lack of syntax highlighting). It 5 s the same Python interpreter, after ali. 

Although this interaction with the interpreter was successful, it only worked 
because we started off in a folder that contained the vsearch.py file. 
Doing this makes this folder the current working directory. Based on how the 
interpreter searches for modules, we know that the current working directory 
is searched first, so it shouldn’t surprise us that this interaction worked and 
that the interpreter found our module. 


But what happens if our module isn’t in the current 
working directory? 
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no import here 


Not Fouwd Modules Produce IwportErrors 

Repeat the exercise from the last page, after moving out of the folder that contains 
our module. Let’s see what happens when we try to import our module now. Here 
is another interaction with the Windows command prompt: 



£tav-t 

Python 

a<xa'm. 

V ^ 


Tvy t O imfovt 
•the module--- 


•••but this 
time we Aet 
ah ev*\ro\rf 


^ , Twiacr Uh this £.3sc, 

we are n,oving {o ihe ^ p _| eve| Q dey) 


File Edit Window Help Redmond #2 


C:\Users\Head First> cd \ 

C:\>py -3 

Python 3.4.3 (v3.4.3:9b73fIc3e601, Feb 24 2015, 22:43:06) [MSC 
v.1600 32 bit (Intel)] on win32 

Type "help", "Copyright", "credits" or "license" for more information. 

»> import vsearch 

Traceback (most recent call last): 

File "<stdin>", line 1, in <module> 

ImportError: No module named 'vsearch 1 
»> quit() 

C:\> 


■ 




The vsearch. py hle is no longer in the interpreter’s current working directo ry, 
as we are now working in a folder other than mymodules. This means our 
module file can’t be found, which in turn means we can’t import it—hence the 
ImportError from the interpreter. 


If we try the same exercise on a platform other than Windows , we get the same 
results (whether we’re on Linux , Unix , or Mac OS X). Here 5 s the above interaction 
with the interpreter from within the mymodules folder on OSX: 


Charae 'm*to ‘the 

•folder and then type 
"^ython^ to s-tavt 

the interpreter- 

|mpo\rt the -_ 

module- 

|*t wovks- we 
ean use the 
module s 

-fuhdtiOhS. 



| File Edit Window Help Cupertino #1 


$ cd mymodules 
mymodules$ python3 

Python 3.4.3 (v3.4.3:9b73fIc3e601, Feb 23 2015, 02:52:03) 

[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin 

Type "help", "Copyright", "credits" or "license" for more information. 
»> import vsearch 

»> vsearch. search4vowels (’ hitch-hiker') 

{'i', 'e'} 

»> vsearch. search41etters (' galaxy' , ' xyz ') 

{'x l , ’ y' } 

»> quit() 

mymodules$ 


V- E*'t the python interpreter and retum to your operatin* 
system s dommdnd p\ro mpt- 
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IwportErrors Occur No Matter the 
Platform 

If you think running on a non -Windows platform will somehow fix this import 
issue we saw on that platform, think again: the same ImportError occurs on module 

UNIX-like Systems, once we change to another folder: 



£*tav*t 
Python 3 
again. 



Chan 9 e i 0 Jnolher 4Ide*- 4 4is case, 

WC a " e rnovin 9 <*<r 4p-lev e | folder). 


Tvy *to irwfoV"t 
*thc module-- 


• bu-t this 
iir*e w t Act 
an c\r\ro\r' 





File Edit Window Help Cupertino #2 


mymodules$ cd 
$ python3 

Python 3.4.3 (v3.4.3:9b73fIc3e601, Feb 23 2015, 02:52:03) 

[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin 

Type "help", "Copyright", "credits" or "license" for more Information 

»> import vsearch 

Traceback (most recent call last): 

File "<stdin>", line 1, in <module> 

ImportError: No module named 'vsearch' 

»> quit() 


As was the case when we were working on Windows , the vsearch.py file is no 
longer in the interpreter’s current working directory, as we are now working in a 
folder other than mymodules. This means our module file can’t be found, which 
in turn means we can’t import it—hence the ImportError from the interpreter. 
This problem presents no matter which platform you’re running Python on. 

- tere, are nQ — 

Dumb Questi^ns 


Can’t we be location specific and say something like import C: \mymodules\vsearch on Windows platforms, or 
perhaps import /mymodules/vsearch on UNIX-like Systems? 

No, you cani Granted, doing something like that does sound tempting, but ultimately won’t work, as you can’t use paths in this way with 
Pythoni import statement. And, anyway, the last thing you’ll want to do is put hardcoded paths into any of your programs, as paths can 
often change (for a whole host of reasons). It is best to avoid hardcoding paths in your code, if at ali possible. 



If I can’t use paths, how can I arrange for the interpreter to find my modules? 


If the interpreter cani find your module in the current working directory, it looks in the site-packages locations as well as in the Standard 
library (and there’s more about site-packages on the next page). If you can arrange to add your module to one of the site-packages locations, 
the interpreter can then find it there (no matter its path). 
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install into Python 


(retting a Module into Site-packages 

Recall what we had to say about site-packages a few pages back when we 
introduced them as the second of three locations searched by the interpreter’s 
import mechanism: 



Your interpreters site-packages locations 

These are the directories that contain any third-party Python modules 
which you may have installed (including any written by you). 



module 


As the provision and support of third-party modules is Central to Python’s 
code reuse strategy, it should come as no surprise that the interpreter comes 
with the built-in ability to add modules to your Python setup. 

Note that the set of modules included with the Standard library is managed 
by the Python core developers, and this large collection of modules has been 
designed to be widely used, but not tampered with. Specifically, don’t add or 
remove your own modules to/from the Standard library However, adding or 
removing modules to your site-packages locations is positively encouraged, so 
much so that Python comes with some tools to make it straightforward. 


Using "setuptools" to Install into site-packages 

As of release 3.4 of Python, the Standard library includes a module called 
setuptools, which can be used to add any module into site-packages. 
Although the details of module distribution can—initially—appear complex, 
all we want to do here is install vsearch into site-packages, which is 
something setuptools is more than capable of doing in three steps: 


o 

o 


Create a distribution description 

This identifies the module we want setuptools to install. 

Generate a distribution file 

Using Python at the command line, we’ll create a shareable 
distribution file to contain our module 5 s code. 



Install the distribution file 

Again, using Python at the command line, install the distribution 
file (which includes our module) into site-packages. 


Python 3.4 (or 
newer) makes using 
setuptools a hreeze. 
II you aren’t running 
3.4 (or newer), 
consider upgrading. 


Step 1 requires us to create (at a minimum) two descriptive files for our 
module: setup . py and README . txt. Let’s see what’s involved. 
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Creating the Required Setup Files 

If we follow the three steps shown at the bottom of the last page, wedl end up 
creating a distribution package for our module. This package is a single 
compressed file that contains everything required to install our module into 
site-packages. 

For Step 1, Create a distribution description , we need to create two files that wedl 
place in the same folder as our vsearch.py file. We’ll do this no matter 
what platform we’re running on. The first file, which must be called setup . 
py, describes our module in some detail. 

Find below the setup . py file we created to describe the module in the 
vsearch.py file. It contains two lines of Python code: the first line imports 
the setup function from the setuptools module, while the second 
invokes the setup function. 

The setup function accepts a large number of arguments, many of 
which are optional. Note how, for readability purposes, our call to setup 
is spread over nine lines. We’re taking advantage of Python 5 s support for 
keyword arguments to clearly indicate which value is being assigned to which 
argument in this call. The most important arguments are highlighted; the first 
names the distribution, while the second lists the . py files to include when 
creating the distribution package: 


□ 

Create a distribution 
description. 

□ 

Generate a 
distribution file. 

□ 

Install the 
distribution file. 


/ 

We II dhedk otf eaeh 
p»>ple-ted step as we Work 
thv-ough this material. 


/">po<rt the "setup" 
tundtion fv-or» the 
"setuptools” module. 


This is an invodation ot 
the "setup” lundtion. — 
WeVe spveading its 
av-ftuments ovev many 
lines. 


from setuptools import setup 

r setup( 

name='vsearch', 


1 


The "name” av-gument 
identi-fies the distribution. |t's 
dommon pv-adtide to name the 
distv-ibution a-Pter the module- 

version='1.0', 

description='The Head First Python Search Tools 
author='HF Python 2e', 
author_email='hfpy2e@gmail.com', 
url='headfirstlabs.com', 


py_modu 1 e s = [' vsearch' ] , 


k ) 


In addition to setup .py, the setuptools mechanism requires the 
existence of one other file—a “readme” file—into which you can put a 
textual description of your package. Although having this file is required, 
its contents are optional, so (for now) you can create an empty file called 
README . txt in the same folder as the setup . py file. This is enough to 
satisfy the requirement for a second file in Step 1. 


Tte « a W of V mcM ' 

** fV tw» «W- « “ 

havc oy \& 
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Creating the Pistributiow File 

At this stage, you should have three files, which we have put in our 
mymodules folder: vsearch . py, setup . py, and README . txt. 

We’re now ready to create a distribution package from these files. This is Step 
2 from our earlier list: Generate a distribution file. We’ll do this at the command 
line. Although doing so is straightforward, this step requires that different 
commands be entered based on whether you are on Windows or on one of the 
UNIX-like operating systems (Linux, Unix, or Mac OS X). 


0 

Create a distribution 
description. 

□ 

Generate a 
distribution file. 

□ 

Install the 
distribution file. 


Creating a distribution file on Windows 


Ku* Pytho» 3 

° h Mhdows. 


If you are running on Windows, open a command prompt in the folder that 
contains your three hies, then enter this command: 

C:\Users\Head First\mymodules> py -3 setup.py sdist 



The Python interpreter goes to work immediately after you issue this 
command. A large number of messages appear on screen (which we show 
here in an abridged form): 


ExeCuie the dode 
i» Vtup.py"... 



... and ?a ss 
"sdist” as a» 


3Y-<yAwevr' 


running sdist 
running egg_info 
creating vsearch.egg-info 


creating dist 

creating 'dist\vsearch-l.0.zip' and adding 'vsearch-1.0' to it 
adding 'vsearch-1.0\PKG-INFO' 
adding ’vsearch-1.0 \README .txt' 


adding ’vsearch-1.0\vsearch.egg-info\top_level.txt' 
removing 'vsearch-1.0' (and everything under it) 


When the Windows command prompt reappears, your three hies have 
been combined into a single distribution file. This is an installable hle 
that contains the source code for your module and, in this case, is called 
vsearch-1.0.zip. 

You’11 hnd your newly created ZIP hle in a folder called dist, which has also 
been created by setuptools under the folder you are working in (which is 
mymodules in our case). 


thetk -that 

y°y e ’ rwmih 9 tast 

Python 3.f, and also 
»ake suire y our "setup. 
Py +ile is identidal to 


ours 
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PistributioH Files on UNIX-like OSes 

If you are not working on Windows , you can create a distributiori file in much 
the same way as on the previous page. With the three files (setup . py, 
README . txt, and vsearch.py)ina folder, issue this command at your 
operating systenfs command line: 


0 

Create a distribution 
description. 

□ 

Generate a 
distribution file. 

□ 

Install the 
distribution file. 


mymodules$ 



R>*r> Pytho* 3. 


python3 setup.py sdist 



Exetefe the toAt 
'i setup.py"... 



...and 

"sdist 3n 


Like on Windows , this command produces a slew of messages on screen: 

running sdist 
running egg_info 
creating vsearch.egg-info 
• • • 

running check 
creating vsearch-1.0 

creating vsearch-1.0/vsearch.egg-info 


creating dist 
Creating tar archive 
removing 'vsearch-1.0' 


(and everything under it) 


When your operating systenfs command line reappears, your three files have 
been combined into a source distribution file (hence the sdist argument 
above). This is an installable file that contains the source code for your 
module and, in this case, is called vsearch-1.0 . tar . gz. 

You’11 find your newly created archive file in a folder called dist, which 
has also been created by setuptools under the folder you are working in 
(which is mymodules in our case). 


TV messaoes di-MW 

Ihose 


wilh IMndows) double- 
cheik eveirylhing. 


With your source distribution file created (as a ZIP or as 
a compressed tar archive), you’re now ready to install 
your module into site-packages. 
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Installing Packages with "pip" 

Now that your distributiori file exists as a ZIP or a tarred archive (depending on your 
platform), it’s time for Step 3: Install the distribution file. As with many such things, 
Python comes with the tools to make this straightforward. In particular, Python 3.4 
(and newer) includes a tool called pip, which is the Package Installer for Python. 


0 

Create a distribution 
description. 

0 

Generate a 
distribution file. 

□ 

Install the 
distribution file. 


Step $ on Windows 

Locate your newly created ZIP file under the di st folder (recall that the file is 
called vsearch-1.0 . zip). While in the Windows Explorer, hold down the Shift 
key, then right-click your mouse to bring up a context-sensitive menu. Select Open 
command window here from this menu. A new Windows command prompt opens. At this 
command prompt, type this line to complete Step 3: 



'7 h e „ odu , e 

I’ p ' fi fi» pip fe install 
the identi+ied 21/p £;| e 


C:\Users\...\dist> py -3 -m pip install vsearch-1.0.zip 


If this command fails with a permissions error, you may need to restart the command 
prompt as the Windows administrator, then try again. 

When the above command succeeds, the following messages appear on screen: 


Processing c:\users\... \dist\vsearch-l .0.zip 
Installing collected packages: vsearch 
Running setup.py install for vsearch 
Successfully installed vsearch-1.0 

-Siittess! 


Step $ on UNIX-like OSes 


On Linux , Unix , or Mac OS X, open a terminal within the newly created dict folder, 
and then issue this command at the prompt: 


Python l with the 
module pip, and then as k 

P‘P "«feli the identitied 

Compiressed tar -file. 



.../dist$ sudo python3 -m pip install vsearch-1.0.tar.gz 


When the above command succeeds, the following messages appear on screen: 

Processing ./vsearch-1.0.tar.gz 
Installing collected packages: vsearch 
Running setup.py install for vsearch 
Successfully installed vsearch-1.0 

-- 

The vsearch module is now installed as part of site-packages. 



P e Ksing the 
sudo" tot*n>a*d hev 
■to ensuve we install 
w 'th the dov^ect 
peirmissions. 
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Modules: What We Kwow Already 

Now that our vsearch module has been installed, we can use import vsearch 
in any of our programs, safe in the knowledge that the interpreter can now find the 
module’s functions when needed. 

If we later decide to update any of the module 5 s code, we can repeat these three steps 
to install any update into site-packages. If you do produce a new version of your 
module, be sure to assign a new version number within the setup . py file. 

Let 5 s take a moment to summarize what we now know about modules: 


0 

Create a distribution 
description. 

0 

Generate a 
distribution file. 

0 

Install the 
distribution file. 


\ 

AH dohc/ 



BULLET POINTS 


■ A module is one or more functions 
saved in a file. 

■ You can share a module by 
ensuring it is always available with 
the interpreter’s current working 
directory (which is possible, but 
brittle) or within the interpreter’s site- 
packages locations (by far the better 
choice). 


Following the setuptools 
three-step process ensures that 
your module is installed into site- 
packages, which allows you to 
import the module and use its 
functions no matter what your current 
working directory happens to be. 


(riving your code away (a.k.a. shariwg) 

Now that you have a distribution file created, you can share this file with other Python 
programmers, allowing them to install your module using pip, too. You can share 
your file in one of two ways: informally, or formally. 

To share your module informally, simply distribute it in whatever way you wish and to 
whomever you wish (perhaps using email, a USB stick, or via a download from your 
personal website). Its up to you, really. 

To share your module formally, you can upload your distribution file to Python’s 
centrally managed web-based Software repository, called PyPI (pronounced “pie- 
pee-eye,” and short for the Python Package Index). This site exists to allow all manner 
of Python programmers to share all manner of third-party Python modules. To 
learn more about what’s on offer, visit the PyPI site at: https://pypi.python.org/ 
pypi. To learn more about the process of uploading and sharing your distribution 
files through PyPI, read the Online guide maintained by the Python Packaging Authority , 
which you’11 find here: https://www.pypa.io. (There 5 s not much to it, but the 
details are beyond the scope of this book.) 

We are nearly done with our introduction to functions and modules. There 5 sjust a 
small mystery that needs our attention (for not more than five minutes). Flip the page 
when you’re ready. 


Any Python 

progframmer 


can 


also 


use 


pip to install 
your module. 


you are here ► 


183 







copy or reference 



The case of the misbehaving function arguments 

Tom and Sarah have just worked through this chapter, and are now arguing 
over the behavior of function arguments. 

Tom is convinced that when arguments are passed into a function, the data 
is passed by value, and he’s written a small function called double to 
help make his case. TonTs double function works with any type of data 
provided to it. 

Here 5 s TonTs code: 


f j\e Mjnute 

Mystery 



def double(arg) 


print('Before: 

'f arg) 

arg = arg * 2 


print('After: 

'f arg) 


Sarah, on the other hand, is convinced that when arguments are passed into 
a function, the data is passed by reference. Sarah has also written a small 
function, called change, which works with lists and helps to prove her point. 

Here 5 s a copy of SaralTs code: 


def change(arg): 

print('Before: ' , arg) 

arg.append('More data') 
print(’After: ' , arg) 


We’d rather nobody was arguing about this type of thing, as—until now— 
Tom and Sarah have been the best of programming buddies. To help resolve 
this, lefs experiment at the >>> prompt in an attempt to see who is right: 

“by value” Tom, or “by reference” Sarah. They can’t both be right, can they? 
Ifs certainly a bit of a mystery that needs solving, which leads to this often- 
asked question: 


Do function arguments support by-value or 
by-reference call semantics in Python? 



In case you need a quick refresher, note that by-value argument passing refers to the practice of using the value of 
a variable in place of a function's argument. If the value changes in the function's suite, it has no effect on the value 
of the variable in the code that called the function. Think of the argument as a copy of the original variable's value. 
By-reference argument passing (sometimes referred to as by-address argument passing) maintains a linkto the 
variable in the code that called the function. If the variable in the function's suite is changed, the value in the code 
that called the function changes, too. Think of the argument as an alias to the original variable. 
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PeiHon$trating Call-by-Value Sewantics 

To work out what Tom and Sarah are arguing about, let’s put their functions into 
their very own module, which we’ll call mystery. py. Here 5 s the module in an 
IDLE edit window: 



These kuo - 

3 VC s*»rv\il 3 V"• ^ 3 ^ 

-takcs 3 smftlc 
av^uwc^t d»*\ > l3'/ s 

OY\ Stxttn, w3^^l3-tcs 

vts valuc, 3*d “bkcn 

dis^bys \{, OY\ strtt* 
3^3‘m- 


mystery, py - /Users/Pau l/Desktop/ NewBook/chQ4/mystery.py (3,5,0) 


def double(arg) : 

print( T Before: arg) 

arg = arg * 2 
printC r After: , arg) 


def change(arg) : 

print( r Before: , arg) 
arg.append( r More data") 
print( r After: , arg) 


<■ 


TWts 

. douklcs -the 
valuc tpassed ’**• 


This -Puhdfioh 
^PpChds a 
sirihg io a*y 
p3ssed i h lisf. 



As soon as Tom sees this module on screen, he sits down, takes control of the 
keyboard, presses F5, and then types the following into IDLE’s >>> prompt. 
Once done, Tom leans back in his chair, crosses his arms, and says: “See? I told 
you it’s call-by-value.” Take a look at TonTs shell interactions with his function: 


»> num = 10 
»> double (num) 
Before: 10 

After: 20 

»> num 
10 


"fow mvokcs 

fhe “double 

fbv-ee 'tiwcs*. ■ 
oy\Cc Wrbb a* 
mtcyv* valuc, 
i\\tY\ Wrlb 3 
stvm^, 3 nd 

-f mslly Wrlb 
a list 


L 


»> saying = 1 Helio ' 

-s> »> double (saying) 

Before: Helio 

After: Helio Helio 

»> saying 
'Helio ' 

»> numbers = [ 42, 256, 16 ] 

P »> double (numbers) 

Before: [42, 256, 16] 

After: [42, 256, 16, 42, 256, 

»> numbers 
[42, 256, 16] 


Eadh invodation don-Pivms 
tbat the valwe passed 
ih as an av-jument is 

fihanged y/i-thin the 

-rundtion s suite, but 
that the value at the 
shell <ren<ains undhan^ed, 
rhat is,: the tundtioh 
airguments appeav to 
don-Pov-m to dall-by-value 
semantids. 


16] 
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over to sarah 


Pewowstratmg Call-by-Referewce Semawtics 

Undeterred by Tom 5 s apparent slam-dunk, Sarah sits down and takes control of the 
keyboard in preparation for interacting with the shell. Here 5 s the code in the IDLE 
edit window once more, with SaralTs change function ready for action: 



The is the ^ 
"mystevyr/’ 
module- 



# O O mystery-py -/Users/PaLil/Desktop/_NawBook/ch04/mystery,py (3.5.0} 

def double(arg) : 

printC T Before: T , arg) , 

arg = arg * 2 > <— r . 

printC T After: T , arg) 

def change(arg) : 

printC T Before: ’, arg) 

arg.append( 1 More data T )V^ — 

printC 'After: T , arg) -fW-tioh 



Ln: ili Coi: 0 


Sarah types a few lines of code into the >>> prompt, then leans back in her 
chair, crosses her arms, and says to Tom: “Well, if Python only supports call- 
by-value, how do you explain this behavior?” Tom is speechless. 

Take a look at SaralTs interaction with the shell: 



Usm^ “the 
list data as To», 

Savah involces 
hev "dhan 9 e" 

•Cundtion- 


»> numbers = [ 42, 256, 16 ] 

»> change (numbers) 

Before: [42, 256, 16] 

After: [42, 256, 16, 'More data'] 

»> numbers 


[42, 256, 16, 


'More data' | 



This is strange behavior. 

Tom’s function clearly shows call-by-value argument semanties, whereas 
Sarah’s function demonstrates call-by-reference. 


took whats happened/ 

This time the avguments 
value has been dhanged in 
the -rundtioh as well as at 
the shell. This would seem 
to su 99 est that python 
tundtions *also* support 
dall-by-reterende semantids. 


How can this be? What’s going on here? Does Python support both? 
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Solved: the case of the misbehaving function arguments 

Do Python function arguments suppor t by-value or by-reference coli semantics? 

Here’s the kicker: both Tom and Sarah are right. Depending on the situation, Pythorks function argument 
semantics support both call-by-value and call-by-reference. 

Recall once again that variables in Python aren’t variables as we are used to thinking about them in other 
programming languages; variables are object references. It is useful to think of the value stored in the 
variable as being the memory address of the value, not its actual value. It’s this memory address that’s passed 
into a function, not the actual value. This means that Python’s functions support what’s more correctly called 
by-object-reference call semantics. 

Based on the type of the object referred to, the actual call semantics that apply at any point in time can differ. 
So, how come in TonTs and SaralTs functions the arguments appeared to conform to by-value and by-reference 
call semantics? First off, they didiTt—they only appeared to. What actually happens is that the interpreter looks 
at the type of the value referred to by the object reference (the memory address) and, if the variable refers to a 
mutable value, call-by-reference semantics apply. If the type of the data referred to is immutable, call-by- 
value semantics kick in. Gonsider now what this means for our data. 

Lists, dictionaries, and sets (being mutable) are always passed into a function by reference— 
any changes made to the variable’s data structure within the function’s suite are reflected in 
the calling code. The data is mutable, after ali. 

Strings, integers, and tuples (being immutable) are always passed into a function by value— 
any changes to the variable within the function are private to the function and are not 
reflected in the calling code. As the data is immutable, it cannot change. 

Which all makes sense until you consider this line of code: 

arg = arg * 2 

How come this line of code appeared to change a passed-in list within the function’s suite, but when the list 
was displayed in the shell after invocation, the list hadn’t changed (leading Tom to believe—incorrectly—that 
all argument passing conformed to call-by-value)? On the face of things, this looks like a bug in the interpreter, 
as weVe just stated that changes to a mutable value are reflected back in the calling code, but they aren’t here. 
That is, TonTs function didnd change the numbers list in the calling code, even though lists are mutable. So, 
what gives? 

To understand what has happened here, consider that the above line of code is an assignment statement. 
Here’s what happens during assignment: the code to the right of the = Symbol is executed first , and then 
whatever value is created has its object reference assigned to the variable on the left of the = Symbol. Executing 
the code arg * 2 creates a new value, which is assigned a new object reference, which is then assigned to the 
arg variable, overwriting the previous object reference stored in arg in the function’s suite. However, the “old” 
object reference stili exists in the calling code and its value hasn’t changed, so the shell stili sees the original list, 
not the new doubled list created in TonTs code. Gontrast this behavior to SaralTs code, which calls the append 
method on an existing list. As there’s no assignment here, there’s no overwriting of object references, so SaralTs 
code changes the list in the shell, too, as both the list referred to in the functions’ suite and the list referred to in 
the calling code have the same object reference. 

With our mystery solved, we’re nearly ready for Ghapter 5. There’s just one outstanding issue. 


plvfc Jy0nufe 

Mystery 
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what about pep 8? 


Caw I Tcst for PEP $ Compliance? 



I have a quick ques+ion before we 
move on. I like the idea of wri+ing 
PEP 8 compliant code...is there any way 
I can automatically check my code for 
compliance? 


t 





Yes. It is possible. 

But not with Python alone, as the 
Python interpreter does not provide 
any way to check code for PEP 8 
compliance. However, there are a 
number of third-party tools that do. 

Before jumping into Ghapter 5, let’s 
take a little detour and look at one 
tool that can help you stay on the 
right side of PEP 8 compliance. 
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(rettiwg Ready to Check PEP 8 Compliance 

Let 5 s detour for just a moment to check our code for PEP 8 compliance. 

The Python programming community at large has spent a great deal of time 
creating developer tools to make the lives of Python programmers a little bit 
better. One such tool is pytest, which is a testingframework that is primarily 
designed to make the testing of Python programs easier. No matter what type 
of tests you’re writing, pytest can help. And you can add plug-ins to pytest to 
extend its capabilities. 

One such plug-in is pep8, which uses the pytest testing framework to check your 
code for violations of the PEP 8 guidelines. 

Kecalling our code 

Let 5 s remind ourselves of our vsearch . py code once more, before feeding it to 
the pytest/pep8 combination to find out how PEP 8-compliant it is. Note that 
weil need to install both of these developer tools, as they do not come installed 
with Python (weil do that over the page). 

One more, here is the code to the vsearch . py module, which is going to be 
checked for compliance to the PEP 8 guidelines: 


DETOUR 


Learn more akout 
jytest Irom 
ittp:/ / doc. pytest. 
orgf/ en/latest/. 


def search4vowels(phrase:str) -> set: 

"""Return any vowels found in a supplied phrase.""" 
vowels = set('aeiou') 

return vowels.intersection(set(phrase)) 




£°de is ih 


def search41etters(phrase:str, letters:str='aeiou') -> set: 
"""Return a set of the 'letters' found in 'phrase'.""" 
return set(letters) .intersection (set (phrase)) 


Installiwg pytest and the pepS plug-in 

Earlier in this chapter, you used the pip tool to install your vsearch. py 
module into the Python interpreter on your computer. The pip tool can also be 
used to install third-party code into your interpreter. 

To do so, you need to operate at your operating systenis command prompt (and 
be connected to the Internet). Youil use pip in the next chapter to install a 
third-party library. For now, though, let 5 s use pip to install the pytest testing 
framework and the pep8 plug-in. 
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Install the Testing Peveloper Tools 

In the example screens that follow, we are showing the messages that appear when you 
are running on the Windows platform. On Windows , you invoke Python 3 using the 
py -3 command. If you are on Linux or Mac OS X, replace the Windows command 
with sudo python3. To install pytest using pip on Windows , issue this command 
from the command prompt while running as administrator (search for cmd. exe, then 
right-click on it, and choose Run as Administrator from the pop-up menu): 

py -3 -m pip install pytest 






DETOUR 


S-tav-t '»* 
mode*** 


••.•then issue -the 
pip domimdnd bo 
install w pytest w ». 


•••■then ehe£k 
whether it 
i^stalled 
sueeesstully. 




Administrator: Command Prompt 


□ 


0 


Microsof tUjy 
Co pyr 


ei‘sion 6.1.7601] 

IJOtr^Slicrosof t Corporation. Ali rights reserued, 


#^Uindous\systen32>py -3 -n pip install pytest 
Collecting pytest 

Doun loading pytest-2.8.7-py2 .py3-none-any .whl <151kB> 
100* ! tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt ! 155kB 1.3MB/S 
Collecting colorana (from pytest> 

Dounloading colorana-0.3.6-py2-py3~none-any.whl 
Collecting py>=1.4.29 (fron pytest> 

Dounloading py-1.4.31-py2.py3-none-any.uhl C81kB> 

100* !tttttttttttttttttttttttttttttttttttttttttttttttttttttttt####! 86kB 131kB/s 
Installing collected packages: colorana, py, pytest 
Successfully installed colorana-0.3.6 py-1.4.31 pytest-2.8 

C: \U 4 o us\s y s t e n3 2 > 


If you examine the messages produced by pip, you’11 notice that two of pytests 
dependencies were also installed (colorama and py). The same thing happens when 
you use pip to install the pep8 plug-in: it also installs a host of dependencies. Here’s 
the command to install the plug-in: 

py -3 -m pip install pytest-pep8 — 


Remember: i-P you aren't 
Jrunnmg Mhdows, replaee 
Vy wih 'sudo python3 w . 


VVViile stili i* 
/\dministvatDV r*odt> 
issue this tommand, 
Y/hith installs the 
plu^-in- 


This doritmj^d 
sueeeeded boo, and 
also installed the 
re^uired dependencies. 


Administrat^- Comma.. J Prompt 

kS? 


0 


J&nu^vJ±ndous\systen32 >py -3 -n pip install pytest-pep8 
Collecting pytest-pep8 

Dounloading pytest-pep8-l.0.6.tar. gs 
Collecting pytest-cache <fron pytest-pep8> 

Dounloading pytest-cache-1.0.tar.gs 

Requirenent already satisfied Cuse —upgrade to upgrade>: pytest>=2.4.2 in c:\pr 
ogran filesXpython 3.5\lib\site-packages (fron pytest-pep8> 

Collecting pep8>=1.3 (fron pytest-pep8> 

Dounloading pep8-l.7.0-py2.py3-none-any.uhl <41kB> 

100* S tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt! 45kB 174kB/s 
Collecting execnet>=l.1.devi Cfron pytest-cache->pytest-pep8> 

Dounloading execnet-1.4.1-py2.py3-none~any.uhl C40kB> 

100* 40kB 174kB/s 

Requirenent already satisfied Cuse —upgrade to upgrade>: py>=1.4.29 in c:\progr 
an filesXpython 3.5\lib\site-packages Cfron pytest>=2.4.2->pytest-pep8> 
Requirenent already satisfied Cuse —upgrade to upgrade>: colorana in c:\progran 
filesVpython 3.5\lib\site-packages Cfron pytest>=2.4.2->pytest-pep8> 

Collecting apipkg>=1.4 Cfron execnet>=1.1.deul->pytest-cache->pytest-pep8> 

Do unlo adin g apipkg-1.4-py2.py3-none-an y.uh1 
Installing collected packages: apipkg, execnet, pytest-cache, pep8, pytest-pep8 
Running setup.py install for pytest-cache ... done 
Running setup.py install for pytest-pep8 ... done 
Successfully installed apipkg—1.4 execnet-1.4.1 pep8-1.7.0 pytest-cache-1.0 pyte 
st-pep8-l.0.6/fl 

C: \U in do us \s y §t e n3 2 > 
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How PEP 8-Cowpliawt Is Our Code? 

With pytest and pep8 installed, you’re now ready to test your code for PEP 8 
compliance. Regardless of the operating system you’re using, youdl issue the same 
command (as only the installation instructions differ on each platform). 

The pytest installation process has installed a new program on your computer 
called py. test. Let’s run this program now to check our vsearch.py code for 
PEP 8 compliance. Make sure you are in the same folder as the one that contains 
the vsearch.py file, then issue this command: 


DETOUR 


py.test --pep8 vsearch.py 

Here 5 s the output produced when we did this on our Windows computer: 


• -tv. i tu» wwurwr 


ta^-t be good, da« i-t? 


C :\Wi n d ows\sy5tem32\cm d. exe 


E:\_NewBook\ch04>py.test —pep8 vsearch.py 
:============================ test session starts === 

platform win32 — Python 3.5.0, pytest-2.8.7, py-1.4.31 J/ pluggy-0.3.1 
rootdir: E:\_NewBook\ch04, inifile: 
plugins: pep8-1.0.6 
collected 1 items 


d || E3 


vsearch.py F 


E:\_NewBook\ch04\vsearch.py :2:25: E231 missing whitespace after 
def search4vouelsCphrase:str> -> set: 


E:\_NewBook\ch04\vsearch.py :3:56: U291 trailing whitespace 
Return any vowels found in a supplied phrase.. 


F8ILURES 


j . j 


IMMPl 


E:\_NewBook\ch04\vsearch.py :7:1: E302 expected 2 blank lines, found 1 
def search41etters<phrase:str, letters:str= J aeiou J > —> set: 

A. 

E:\_NewBook\ch04\vsearch.py :7:26: E231 missing whitespace after J : J 
def s e arc h4le 11 e rs < phras e:s t r, letters:str= J aeiou J > —> set: 

A 

E:\_NewBook\ch04\vsearch.py :?:39: E231 missing whitespace after J : 1 
def search4le 11 e rs < phras e:s t r, letters:str= J aeiou'> —> set: 


E:\_Ne wBo o k\c h04 > 


Whoops! It looks like we have failures, which means this code is not as compliant 
with the PEP 8 guidelines as it could be. 

Take a moment to read the messages shown here (or on your screen, if you are 
following along). All of the “failures” appear to refer—in some way—to whitespace 
(for instance, spaces, tabs, newlines, and the like). Let 5 s take a look at each of them 
in a little more detail. 
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py.test -pep8 rocks 


Undmtanding the Failore Messages 

Together, pytest and the pep8 plug-in have highlighted five issues with our 
vsearch. py code. 

The first issue has to do with the fact that we haven’t inserted a space after the : 
character when annotating our function 5 s arguments, and weVe done this in three 
places. Look at the first message, noting pytest’s use of the caret character ( A ) to 
indicate exactly where the problem is: 


...:2:25: E231 missing whitespace after ':' 
def search4vowels(phrase:str) -> set: 

iVs wv-cm<y 





DETOUR 


ffa res 


Mrohg. 


If you look at the two issues at the bottom of pytests output, you’11 see that 
weVe repeated this mistake in three locations: once on line 2, and twice on line 7. 
There’s an easy fix: add a single space character after the colon. 

The next issue may not seem like a big deal, but is raised as a failure because the 
line of code in question (line 3) does break a PEP 8 guideline that says not to 
include extra spaces at the end of lines: 

...:3:56: W291 trailing 

"""Return any vowels found in a supplied phrase.""" 

A 


IVkati 


whitespace 


2 


wvohg 


Dealing with this issue on line 3 is another easy fix: remove all trailing whitespace. 
The last issue (at the start of line 7) is this: 


. . .7:1: E302 expected 2 blank 

def search41etters(phrase:str, letters:str='aeiou’) -> set: 



This issue pvesenis ai ihe siavi ot li«e "7- 


lines, found 1 


wvoy><y 


There is a PEP 8 guideline that offers this advice for creating functions in a 
module: Surround top-levelfunction and class definitions with two blank lines. In our code, 
the search4vowels and search4letters functions are both at the “top 
level” of the vsearch . py file, and are separated from each other by a single 
blank line. To be PEP 8—compliant, there should be two blank lines here. 

Again, it’s an easy fix: insert an extra blank line between the two functions. Let’s apply 
these fixes now, then retest our amended code. 


BTW: Ckeck out 
http://pep8.org/ fora 
beautifully rencterect 
version of Pythoni 
style guidelines. 
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Confirming FEP $ Cowpliawce 

With the amendments made to the Python code invsearch.py, the file’s 
contents now look like this: 




DETOUR 


def 


def 


search4vowels(phrase: str) -> set: 

"""Return any vowels found in a supplied phrase.""" 
vowels = set('aeiou') 

return vowels.intersection(set (phrase)) 


search41etters(phrase: str, letters: str='aeiou') -> set 
"""Return a set of the 'letters' found in 'phrase'.""" 
return set (letters) .intersection (set (phrase)) 


The PEP li^i 
versioh o-f w vseav-£h.py w 


When this version of the code is run through pytests pep8 plug-in, the output 
confirms we no longer have any issues with PEP 8 compliance. Here’s what we 
saw on our computer (again, running on Windows): 


6(rtt* is good—this tode 
has ho PEP 0 issues. © 


C :\Wi n d o ws\sy stem32\c m d , exe 


E:\_NewBook\ch04>py.test —pep8 usearch.py / 

=============== = = = = = = = = = = = = = = test session starts ====== il ============ 

platform win32 — Python 3.5.0, pytest-2.8.7, py-1.4.31,,/pluggy-0.3.1 
rootdir: E:\_NetiBook\ch04, inifile: I 

plugin s: pe p8-1.0.6 m 

collected 1 itens // 


S II 


use arch.py 


1 passed in 0.06 seconds 


E:\_Ne uBo o k\c h04 > 


Cowforwawce to S is a good thiwg 


If you’re looking at all of this wondering what all the fuss is about (especially over 
a little bit of whitespace), think carefully about why you’d want to comply to PEP 
8. The PEP 8 documentation States that readability counts , and that code is read 
much more often than it is written. If your code conforms to a Standard coding style, it 
follows that reading it is easier, as it “looks like” everything else the programmer 
has seen. Gonsistency is a very good thing. 

From this point forward (and as much as is practical), all of the code in this book 
will conform to the PEP 8 guidelines. You should try to ensure your code does too. 



This is -the 
Chd of -the 
pytes-t deW. 
See you ih 
Chap-tev 5. 
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Chapter 4'$ Code 


def search4vowels(phrase: str) -> set: 

"""Returns the set of vowels found in 'phrase'. MMM 
return set ( 'aeiou') .intersection(set(phrase)) 

def search41etters(phrase: str, letters: str='aeiou') -> set: 
"""Returns the set of 'letters' found in 'phrase'.""" 
return set (letters) .intersection (set(phrase)) 


This is -the dode 
TV-orw -the “vseavdh.py" 
module, whieH £o»vtair,s 
OWr two -Pundtiohs: 
seavdh^vowels” and 
"seavdh4iettevs". 



This is -the "seRf • 
py " -f ile, MjViidV» 
allov/ed us x° 

twvn ouV module 

m to a» mstallable 

distvibution- 



from setuptools import setup 
setup ( 

name='vsearch', 
version='1.0', 

description='The Head First Python Search Tools', 

author='HF Python 2e', 

author_email='hfpy2e@gmail.com', 

url='headfirstlabs.com', 

py modules=['vsearch'], 


def 

double(arg): 
print ( 'Before: ', 

arg) 


arg = arg * 2 
print('After: ', 

arg) 

def 

change(arg: list) 

• 

• 


print('Before: ', 

arg) 


arg.append('More 

data') 


print('After: ', 

arg) 


Md this is the "mystevy.py" 
«•odule, whidh had Tom and 
Sairah upset ai eae h othev. 
Thank+ully, novi that the 
««ystevy is solved, they ave 
badk io being pvogvamming 
buddies ohde move. © 
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At this stage, you know enough Python to be dangerous. 

With this book’s first four chapters behind you, you’re now in a position to productively use 
Python within any number of application areas (even though there’s stili lots of Python to 
learn). Rather than explore the long list of what these application areas are, in this and 
subsequent chapters, we’re going to structure our learning around the development of 
a web-hosted application, which is an area where Python is especially strong. Along the 
way, you’ll learn a bit more about Python. Before we get going, however, let’s have a quick 
recap of the Python you already know. 


this is a new chapter 
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Python: What You Already Know 

Now that you Ve got four chapters under your belt, let’s pause for a moment 
and review the Python material presented so far. 



BULLET POINTS 


■ IDLE, Pythorfs built-in IDE, is used to experiment with 
and execute Python code, either as single-statement 
snippets or as larger multistatement programs written 
within IDLE’s text editor. As well as using IDLE, you 
ran a file of Python code directly from your operating 
system’s command line, using the py -3 command 
(on Windows) orpython3 (on everything else). 

■ YouVe learned how Python supports single-value data 
items, such as integers and strings, as well as the 
booleans True and False. 

■ YouVe explored use cases for the four built-in data 
structures: lists, dictionaries, sets, and tuples. You 
know that you can create complex data structures by 
combining these four built-ins in any number of ways. 

■ YouVe used a collection of Python statements, including 

if, elif, else, return, for, from, and 
import. 

■ You know that Python provides a rich Standard library, 
and youVe seen the following modules in action: 

datetime, random, sys, os, time, html, 
pprint, setuptools, and pip. 

■ As well as the Standard library, Python comes with a 
handy collection of built-in functions, known as the 
BIFs. Here are some of the BIFs youVe worked with: 

print, dir, help, range, list, len, input, 
sorted, dict, set, tuple, and type. 

■ Python supports ali the usual operators, and then some. 
Those youVe already seen include: in, not in, +, 

-, = (assignment), == (equality), +=, and *. 


As well as supporting the square bracket notation for 
working with items in a sequence (i.e., [ ]), Python 
extends the notation to support slices, which allow you 
to specify start, stop, and step values. 

YouVe learned how to create your own custom functions 
in Python, using the def statement. Python functions 
can optionally accept any number of arguments as well 
as return a value. 

Although ifs possible to enclose strings in either single 
or double quotes, the Python conventions (documented 
in PEP 8) suggest picking one style and sticking to it. 

For this book, weVe decided to enclose all of our strings 
within single quotes, unless the string we’re quoting 
itself contains a single quote character, in which case 
well use double quotes (as a one-off, special case). 

Triple-quoted strings are also supported, and youVe 
seen how they are used to add docstrings to your 
custom functions. 

You learned that you can group related functions into 
modules. Modules form the basis of the code reuse 
mechanism in Python, and youVe seen how the pip 
module (included in the Standard library) lets you 
consistently manage your module installations. 

Speaking of things working in a consistent manner, 
you learned that in Python everything is an object, 
which ensures—as much as possible—that everything 
works just as you expect it to. This concept really pays 
off when you start to define your own custom objects 
using classes, which well show you how to do in a later 
chapter. 
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let'$ Poild Something 


0 



y 




LeVs build a webapp. 

Specifically, let’s take our search41etters 
function and make it accessible over the Web, 
enabling anyone with a web browser to access 
the Service provided by our function. 

We could build any type of application, but 
building a working web application lets us 
explore a number of Python features while 
building something that’s generally useful, as 
well as being a whole heap meatier than the 
code snippets youVe seen so far in this book. 

Python is particularly strong on the server side 
of the Web, which is where we’re going to 
build and deploy our webapp in this chapter. 

But, before we get going, let’s make sure 
everyone is on the same page by reviewing 
how the Web works. 
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how the web works 



Vebapps XJp C]<sse - 

No matter what you do on the Web, it’s all about requests and responses. A web request is sent 
from a web browser to a web server as the resuit of some user interaction. On the web server, a 
web response (or reply ) is formulated and returned to the web browser. The entire process can 
be summarized in five steps, as follows: 


Step 1: Your user enters 
a web address, clicks 
a hyperlink, or clicks a 
button in her chosen 
web browser. 


Step 2: The web 
browser converts 
the user’s action 
into a web request 
and sends itto a 
server over the 
Internet. 



Step 3: The web server 
receives the web request 
and has to decide what 
to do next... 


Web 

Server 




0 t 


Peciding what to do next 

One of two things happen at this point. If the web request 
is for static content —such as an HTML file, image, or 
anything else stored on the web server’s hard disk—the web 
server locates the resource and prepares to retura it to the web 
browser as a web response. 

If the request is for dynamic content —that is, content that 
must be generated , such as search results or the current contents 
of an online shopping basket—the web server runs some code 
to produce the web response. 
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The (potewtially) wawy substeps of Step $ 

In practice, Step 3 can involve multiple substeps, depending 
on what the web server has to do to produce the response. 
Obviously, if all the server has to do is locate static content 
and retura it to the browser, the substeps aren’t too taxing, as 
it’s just a matter of reading from the web server’s disk drive. 

However, when dynamic content must be generated, the 
substeps involve the web server running code and then 
capturing the output from the program as a web response, 
before sending the response back to the waiting web browser. 



Step 4: The web server 
sends the response 
back over the Internet 
to the waiting web 
browser. 



Step 5: The web 
browser receives the 
web response and 
displays iton your 
user’s screen. 
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the webapp spec 


What Po We Wawt Our Webapp to Po? 


As tempting as it always is to just start coding , let 5 s first think about how our 
webapp is going to work. 


Users interact with our webapp using their favo rite web browser. All they 
have to do is enter the URL for the webapp into their browser 5 s address bar 
to access its Services. A web page then appears in the browser asking the user 
to provide arguments to the search4letters function. Once these are 
entered, the user clicks on a button to see their results. 

Recall the def line for our most recent version of search41etters, 
which shows the function expecting at least one—but no more than two— 
arguments: a phrase to search, together with the letters to search for. 
Remember, the letters argument is optional (defaulting to aeiou): 


The "det" line tor the 

l^htletW' Wtion, 
w ^h takes one, but no 
*** ^ two, avguments 



def search41etters(phrase:str, letters:str= 1 aeiou 1 ) -> set: 


Let’s grab a paper napkin and sketch out how we want our web page to 
appear. Here 5 s what we came up with: 


1 



One ihpu-fc box 
bas \roor* -fov~ the 
phrase , while 
dhother allows 
tbe 'letters' to 
be cttered (noie 
*tbe de-fault). 











building a webapp 


What Happcws ow the Web Server? 

When the user clicks on the Do it! button, the browser sends the data to the waiting 
web server, which extracts the phrase and letters values, before calling the 
search41etters function on behalf of the now-waiting user. 

Any results from the function are returned to the user 5 s browser as another web page, 
which we again sketeh out on a paper napkin (shown below). For now, let’s assume the 
user entered “hitch-hiker” as the phrase and left the letters value defaulted to 
aeiou. Here 5 s what the results web page might look like: 


ttelre are yowr *- tS ul£s- 



Other than the knowledge you already have about Python, the only thing you need 
to build a working server-side web application is a web application framework, 
which provides a set of general foundational technologies upon which you can build 
your webapp. 

Although it’s more than possible to use Python to build everything you need from 
scratch, it would be madness to contemplate doing so. Other programmers have 
already taken the time to build these web frameworks for you. Python has many 
choices here. However, we’re not going to agonize over which framework to choose, 
and are instead just going to pick a popular one called Flash and move on. 


you are here ► 


201 









get the tools 


Let s Install Flask 


We know from Ghapter 1 that Pythoni Standard library comes with lots of batteries 
included. However, there are times when we need to use an application-specific third- 
party module, which is not part of the Standard library Third-party modules are 
imported into your Python program as needed. However, unlike the Standard library 
modules, third-party modules need to be installed before they are imported and used. 
Flask is one such third-party module. 

As mentioned in the previous chapter, the Python community maintains a centrally 
managed website for third-party modules called PyPI (short for the Python Package 
Index) , which hosts the latest version of Flask (as well as many other projects). 

Recall how we used pip to install our vsearch module into Python earlier in this 
book. pip also works with PyPI. If you know the name of the module you want, you 
can use pip to install any PyPI-hosted module directly into your Python environment. 


Finct PyPI at 


pypi.pytkon.org 


Install Flask from the commamHme with pip 

If you are running on Linux or Mac OS X, type the following command into a terminal 
window: 

$ sudo -H python3 -m pip install flask 



Use this 

dommand Oh /l/|a£ 

OS )< ahd Lihux. 


If you are running on Windows , open up a command prompt—being sure to Run as 
Administrator (by right-clicking on the option and choosing from the pop-up menu)— 
and then issue this command: 

C:\> py -3 -m pip install flask 


This command (regardless of your operating system) connects to the PyPI website, 
then downloads and installs the Flask module and four other modules Flask depends 

on: Werkzeug, MarkupSafe, Jinja2, and itsdangerous. Doni worry (for now) 
about what these extra modules do; just make sure they install correctly. If all is well, 
youil see a message similar to the following at the bottom of the output generated by 
pip. Note that the output runs to over a dozen lines or so: 



Hoic’ Cdsc is 
iimportaht here- 
Thats a I ov/crcssc 
T U tt *flask w . 


Use -this 
eoh^hx^hd Oh 
Mhdows. 


Successfully installed Jinja2-2.8 MarkupSafe-0.23 Werkzeug-0.11 
itsdangerous-0.24 


flask-0.10.1 


nr 


If you doni see the “Successfully installed. . . ” message, make sure youie 
connected to the Internet, and that you Ve entered the command for your operating 
system exactly as shown above. And doni be too alarmed if the version numbers for 
the modules installed into your Python differ from ours (as modules are constantly 
being updated, and dependencies can change, too). As long as the versions you install 
are at least as current as those shown above, everything is fine. 


At the tir*e o-p 
wH-tihg, these are 
the £urreht versioh 
numbers assoeiated 

'with these modules. 
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How Pocs Flask Work? 


Flask provides a collectiori of modules that help you build server-side web 
applicatioris. It’s technically a micro web framework, in that it provides the 
minimum set of technologies needed for this task. This means Flask is not as 
feature-full as some of its competitors—such as Django, the mother of all 
Python web frameworks—but it is small, lightweight, and easy to use. 

As our requirements aren’t heavy (we only have two web pages), Flask is more 
than enough web framework for us at this time. 

Check that Flask is iwstalled and workiwg 

Here 5 s the code for the most basic of Flask webapps, which we are going to 
use to test that Flask is set up and ready to go. 

Use your favo rite text editor to create a new file, and type the code shown 
below into the file, saving it is as hello_f lask. py (you can save the file in 
its own folder, too, if you like—we called our folder webapp): 



Django is a hugely popular 
web applicatiori framework 
within the Python community. 
It has an especially strong, 
prebuilt administration 
facility that can make working 
with large webapps very 
manageable. It's overkill for 
what we're doing here, so 
we've opted for the much 
simpler, but more lightweight, 
Flask. 


This is 
Vllo_ 
•Claskp/"- 



@app.route ( ' / ' ) 
def hello() -> str: 

return 'Helio world from Flask!' 


app.run () 



Type this todc 
exadtly as 
shovim heve— 


well get to 
what it means in 


a moment- 


I Reapy Batte 






from flask import Flask 


app = Flask( name ) 


Run Flask from your OS command line 

Don’t be tempted to run this Flask code within IDLE, as IDLE wasn’t really 
designed to do this sort of thing well. IDLE is great for experimenting with 
small snippets of code, but when it comes to running applications, you 
are much better off running your code directly via the interpreter, at your 
operating systenTs command line. Let’s do that now and see what happens. 


Don’t use IDLE 

to run this code. 
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time for flask 


Ruwmwg Your Flask Webapp for the First Time 

If you are running on Windows , open a command prompt in the folder that contains your 
he Ilo flask.py program file. (Hint: if you have your folder open within the File 
Explorer , press the Shift key together with the right mouse button to bring up a context- 
sensitive menu from which you can choose Open command window here). With the Windows 
command line ready, type in this command to start your Flask app: 


Vte saved ouv- 

todt m a -foldev 

dalled “webapp • 


C:\webapp> py -3 hello flask.py 





If you are on Mac OS X or Linux , type the following command in a terminal window. Be 
sure to issue this command in the same folder that contains your hello_f lask. py 
program file: 

$ python3 hello flask.py 


Asks -fche Py-fch oh 

•fiieirpveiev' {# 
^h -the todt ih 
bello__-Plask.py/ ; 


No matter which operating system you’re running, Flask takes over from this point 
on, displaying status messages on screen whenever its built-in web server performs any 
operation. Immediately after starting up, the Flask web server confirms it is up and running 
and waiting to Service web requests at FlasUs test web address (12 7.0.0.1) and protocol 
port number (50 0 0): 

* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) 


/t you see -this 

ali is well. 




Flask 5 s web server is ready and waiting. JVow what? Lef s interact with the web server using 
our web browser. Open whichever browser is your favorite and type in the URL from the 
Flask web server’s opening message: 


http://127.0.0.1:5000/ 


After a moment, the “Hello world from Flask!” message from hello flask.py should 
appear in your browser’s window. In addition to this, take a look at the terminal window 
where your webapp is running...a new status message should Ve appeared too, as follows: 


This is the address 

«here y<x*r webafT 

is vuvwifi^' r^ter 
it e*3ttly as 
showh bcV"C- 


* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) 
127.0.0.1 - - [23/Nov/2015 20:15:46] "GET / HTTP/1.1" 200 


M ha/ 

Somcthihg 

happehed. 




Getting into the specifics of what constitutes a protocol port number is beyond the scope of this 
book. However, if yoifd like to know more, start reading here: 

https://en.zvikipedia.org/zviki/Port_(computer_netzvorking) 
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Here $ What Happened (Une by Line) 

In addition to Flask updating the terminal with a status line, your web browser 
now displays the web server 5 s response. Here 5 s how our browser now looks (this is 
Safari on Mac OS X)\ 


There s -the 
message 
ve-tiA-ned -fv-om 
-the Flask web 



sevvev. 


• • • < 


Helio world from Flask! 



QO * 127.0.0.1:5000 




d 


& 


+ 


By using our browser to visit the URL listed in our webapp’s opening status 
message, the server has responded with the “Helio world from Flask!” message. 

Although our webapp has only six lines of code, there’s a lot going on here, so 
lef s review the code to see how all of this happened, taking each line in turn. 
Everything else we plan to do builds on these six lines of code. 

The first line imports the Flask class from the flask module: 



This is -the 
£lass 

Flask” wi-fch ah 
uppe\r£ase U P. 


Remember when we discussed alternate ways of importing? 

You could have written import flask here, then referred to the Flask class 
as flask.Flask, but using the from version of the import statement in this 
instance is preferred, as the flask. Flask usage is not as easy to read. 
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getting into flask 


Creatmg a Flask Webapp Object 

The second line of code creates an object of type Flask, assigning it to 
the app variable. This looks straightforward, but for the use of the strange 
argument to Flask, namely_ name_: 



Ma-ts -the deal here? 


The_ name _value is maintained by the Python interpreter and, when 

used anywhere within your progranTs code, is set to the name of the currently 
active module. It turns out that the Flask class needs to know the current 

value of_ name _ when creating a new Flask object, so it must be 

passed as an argument, which is why weVe used it here (even though its usage 
does look strange). 

This single line of code, despite being short, does an awful lot for you, as the 
Flask framework abstracts away many web development details, allowing you 
to concentrate on defining what you want to happen when a web request 
arrives at your waiting web server. We do just that starting on the very next 
line of code. 



Note that_ name _is two underscore characters followed by the word // name"followed by another 

two underscore characters, which are referred to as "double underscores" when used to prefix and 
suffix a name in Python code. You'll see this naming convention a lot in your Python travels, and 
rather than use the long-winded: "double underscore, name, double underscore" phrase, savvy Python 
programmers say: "dunder name," which is shorthand for the same thing. As there's a lot of double 
underscore usages in Python, they are collectively known as "the dunders," and you'll see lots of 
examples of other dunders and their usages throughout the rest of this book. 

As well as the dunders, there is also a convention to use a single underscore character to prefix certain 
variable names. Some Python programmers refer to single-underscore-prefixed names by the groan- 
inducing name "wonder" (shorthand for"one underscore"). 
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Pecoratiwg a Fuwctiow with a URI 

The next line of code introduces a new piece of Python syntax: decorators. 
A function decorator, which is what we have in this code, adjusts the behavior 
of an existing function without you having to change that function’s code (that 
is, the function being decorated). 

You might want to read that last sentence a few times. 

In essence, decorators allow you to take some existing code and augment 
it with additional behavior as needed. Although decorators can also be 
applied to classes as well as functions, they are mainly applied to functions, 
which results in most Python programmers referring to them as function 
decorators. 

Let 5 s take a look at the function decorator in our webapp’s code, which is easy 
to spot, as it starts with the @ symboli 



Geejc Bifi 


PythorYs decorator syntax 
take inspiration from Java's 
annotation syntax, as well 
as the world of functional 
programming. 


tteve s ihc 

decorata*-, v^Vii^V»—likc ali 
dedovaWs—is pveti*ed 
wtb -tbe @ symboI- \ 



Although it is possible to create your own function decorators (coming up in a 
later chapter), for now let’s concentrate on just using them. There are a bunch 
of decorators built in to Python, and many third-party modules (such as 
Flask) provide decorators for specific purposes (route being one of them). 


FlasUs route decorator is available to your webapp’s code via the app 
variable, which was created on the previous line of code. 

The route decorator lets you associate a URL web path with an existing 
Python function. In this case, the URL “/” is associated with the function 
defined on the very next line of code, which is called hello. The route 
decorator arranges for the Flask web server to call the function when a 
request for the “/” URL arrives at the server. The route decorator then 
waits for any output produced by the decorated function before returning the 
output to the server, which then returns it to the waiting web browser. 

It’s not important to know how Flask (and the route decorator) does all of 
the above “magic.” What is important is that Flask does all of this for you, 
and all you have to do is write a function that produces the output you require. 
Flask and the route decorator then take care of the details. 


A lunction decorator 
adjusts tke kekavior 
of an existing lunction 
(witkout ckanging tke 
Iunction’s code). 
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up and running 


Running Your Webapps Pehavior(s) 

With the route decorator line written, the function decorated by it starts on the 
next line. In our webapp, this is the hello function, which does only one thing: 
returns the message “Hello world from Flask!” when invoked: 


TWis is ysi a ve^ulav 

Python -Cunttion 
v/hidh, when invoked) 
vetuvns a stv"in^ to 
its dallcv- (note the 

U> s bc' annotation). 



from flask import Flask 

app = Flask(_name_) 

@app.route ( ' / ' ) 
def hello() -> str: 

return 'Hello world from Flask!' 

app.run () 


The final line of code takes the Flask object assigned to the app variable and asks 
Flask to start running its web server. It does this by invoking run: 


from flask import Flask 

app = Flask(_name_) 

@app.route ( '/') 
def hello() -> str: 

return 'Hello world from Flask!' 

app . run () _ _ _ 


At this point, Flask starts up its included web server and runs your webapp code 
within it. Any requests received by the web server for the “/” URL are responded 
to with the “Hello world from Flask!” message, whereas a request for any other 
URL results in a 404 “Resource not found” error message. To see the error 
handling in action, type this URL into your browser’s address bar: 


Asks -the 
webapp 
io sia\ri 

Running 


http://127.0.0.1:5000/doesthiswork.html 

Your browser displays a “Not Found” message, and your webapp running within 
its terminal window updates its status with an appropriate message: 

* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) 

127.0.0.1 - - [23/Nov/2015 20:15:46] "GET / HTTP/1.1" 200 - 
127.0.0.1 - - [23/Nov/2015 21:30:26] "GET /doesthiswork.html HTTP/1.1" 


That URL 

does not 


exis-fc: 


404 - 




Tbe messages you see *>ay dittev slightly. 
Don t let this y/o\rvy you. 
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Exposing Functionality to the Web 


Putting to one side the fact that youVe just built a working webapp in a mere 
six lines of code, consider what Flask is doing for you here: it’s providing a 
mechanism whereby you can take any existing Python function and display its 
output within a web browser. 


To add more functionality to your webapp, all you have to do is decide on the 
URL you want to associate your functionality with, then write an appropriate 
@app . route decorator line above a function that does the actual work. 

Let 5 s do this now, using our search4 letters functionality from the last 
chapter. 




Let's amend hello_f lask. py to include a second URL: / search4. 
Write the code that associates this URL with a function called do_ 
search, which calls the search41etters function (from our 
vsearch module).Then arrange for the do_search function to return 
the results determined when searching the phrase: "life, the universe, and 
everything!"for this string of characters: 'eiru, ! '. 

Shown below is our existing code, with space reserved for the new code 
you need to write. Your job is to provide the missing code. 


po you 
need “to 
iw^ov-t 
awytWmj? 


Hint: the results returned from search41etters are a Python set. Be 
sure to cast the results to a string by calling the str BIF before returning 
anything to the waiting web browser, as it's expecting textual data, not a 
Python set. (Remember:"BIF" is Python-speak for built-in function.) 

from flask import Flask 


app = Flask( name ) 


Add m a 

se£oy\d 

dedova-tov-. 


@app.route( ' / ' ) 
def hello() -> str: 

return 'Helio world from Flask!' 



app.run () 


/\dd toAe tor 

-the w do__seavth” 
-fufi^bo* heve- 
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doing a do_search 


Wr / «7" 

i v Solution 


You were to amend hello_f lask. py to include a second URL, 
/search4, writing the code that associates the URL with a function 
called do_search, which itself calls the search41etters 
function (from our vsearch module). You were to arrange for 
the do_search function to return the results determined when 
searching the phrase:"life, the universe, and everything!"for the 
string of characters: 'eiru, ! ! . 


You need to 
import the 

U sear£h / Hetters , 

^undtion «fv-ow» the 
w vseareh” r*odule 
be-fore you 


Shown below is our existing code, with space reserved for the new 
code you need to write. Your job was to provide the missing code. 

How does your code compare to ours? 


from flask import Flask 


trom vseardh import seardh^Hetters 


f\ sedond 
dedorator scts 

up the Vseardh^r 

URL 


app = Flask(_name_) 

The u dojscar6b" 

@app. route (' / ' ) Wtion mvokcs 

def hello() -> str: 'WMIetters", 

return 'Helio world from Flask!' , , ^ 

then retums any 

.. . . ; . results as a string. 

@approute( /seardhT ) 

de-P dojseardhO -> str-- 

return strfseardh^lettersOi-Pe, the universe, and everything, Viru/)) 



app.run () 


To test this new functionality, you’11 need to restart your Flask webapp, as it is currently running 
the older version of your code. To stop the webapp, return to your terminal window, then press 
Ctrl and G together. Your webapp will terminate, and you’11 be returned to your operating 
system’s prompt. Press the up arrow to recall the previous command (the one that previously 
started hello_f lask. py) and then press the Enter key. The initial Flask status message 
reappears to confirm your updated webapp is waiting for requests: 


s top th e 

webapp... 



$ python3 heilo_flask.py 

* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) 

127.0.0.1 - - [23/Nov/2015 20:15:46] "GET / HTTP/1.1" 200 - 
127.0.0.1 - - [23/Nov/2015 21:30:26] "GET /doesthiswork.html HTTP/1.1" 
A C 


404 - 


...then / 
v-estavt vt- 


$ python3 heilo_flask.py 
* Running on http://127.0.0.1:5000/ 


(Press CTRL+C to quit) 


Wc are up and 
running again. 
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building a webapp 



As you haverTt changed the code associated with the default 7 ’ URL, that functionality stili works, 
displaying the “Helio world from Flask!” message. 

However, if you enter http://127. O.O.l:5000/search4 into your browser’s address bar, you’ll 
see the results from the call to search4letters: 


Theve ave ‘the vesults 
-('vow' “the eall “to 

VaveV^lettevs". 
^va^-ted, “tViis 
is v\o*thur\^ “to ftet 
eit&Vkcd about bu*t rt 
does pvove “that usmj 
the Yscarth^ WRL 
mvokes the -(Wticm 
a*d vetias the 
vcsults- 




tueret are no 

Diimb QuestiQns 


rm a little confused by the 12 7.0.0.1 and : 5 0 0 0 
parts of the URL used to access the webapp. What’s the deal 
with those? 

At the moment, you’re testing your webapp on your computer, 
which—because it’s connected to the Internet—has its own unique 
IP address. Despite this fact, Flask doesn’t use your IP address 
and instead connects its test web server to the Internet^ loopback 
address: 127.0.0.1, also commonly known as localhost. 
Both are shorthand for “my computer, no matter what its actual 
IP address is.” For your web browser (also on your computer) to 
communicate with your Flask web server, you need to specify the 
address that is running your webapp, namely: 12 7.0.0.1. This is 
a Standard IP address reserved for this exact purpose. 

The : 5000 part of the URL identifies the protocol port number 
your web server is running on. 


Typically, web servers run on protocol port 8 0, which is an Internet 
Standard, and as such, doesn’t need to be specified. You could type 
oreilly. com: 8 0 into your browser’s address bar and itwould 
work, but nobody does, as oreilly. com alone is sufficient (as 
the : 80 is assumed). 

When you’re building a webapp, it’s very rare to test on protocol 
port 80 (as that’s reserved for production servers), so most web 
frameworks choose another port to run on. 8 0 8 0 is a popular 
choice for this, but Flask uses 5 0 0 0 as its test protocol port. 

O: Can I use some protocol port other than 5000 when I test 
and*run my Flask webapp? 

Yes, app. run () allows you to specify a value for port 
that can be set to any value. But, unless you have a very good 
reason to change, stick with Flask’s default of 5 0 0 0 for now. 
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what we’re doing 


Recall What We 're Tryiwg to Puild 

Our webapp needs a web page which accepts input, and another which displays the 
results of feeding the input to the search41etters function. Our current webapp 
code is nowhere near doing all of this, but what we have does provide a basis upon 
which to build what is re quire d. 

Shown below on the left is a copy of our current code, while on the right, we have 
copies of the “napkin specifications” from earlier in this chapter. We have indicated 
where we think the functionality for each napkin can be provided in the code: 

. . ■ 
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building a webapp 


Puilding the HTML Form 

The required HTML form isrft all that complicated. Other than the 
descriptive text, the form is made up of two input boxes and a button. 

Put...what if youVe wew to all this HTML stoff? 

Don’t panic if all this talk of HTML forms, input boxes, and buttons has you 
in a tizzy. Fear not, we have what you’re looking for: the second edition of 
Head First HTML and CSS provides the best introduction to these technologies 
should you require a quick primer (or a speedy refresher). 



Even if the thought of setting aside this book in order to bone up on HTML 
feels like too much work, note that we provide all the HTML you need to 
work with the examples in the book, and we do this without you having to be 
an HTML expert. A little exposure to HTML helps, but it 5 s not a absolute 
requirement (after all, this is a book about Python, not HTML). 

Create the HTML, thew sewd It to the browser 

There 5 s always more than one way to do things, and when it comes to 
creating HTML text from within your Flask webapp, you have choices: 


No-te 4or« /Warketma: 
This is -the book 
we wholehear-tedly 
'reeomme^d -Pov- ^utekly 
getfcihg up io speed 
wi-fch WT/WL...ho-fc 
tha-fc weVe biased or 
dhy-fcbihg. © 


I like to put my HTML 
inside large strings, which I then 
embed in my Python code, returning 
the strings as needed. That way, 
everything I need is right there in my 
code, and I have complete control... 
which is how I roll. Whafs not to 
like, Laura? 



Well, Bob, putting all the HTML in 
your code works, but it doesn't scale. 
As your webapp gets bigger, all that 
embedded HTML gets kinda messy... 
and it's hard to hand of f your HTML 
to a web designer to beautify. Nor 
is it easy to reuse chunks of HTML. 
Therefore, I always use a template 
engine with my webapps. Ifs a bit 
more work to begin with, but over 
time I f ind using templates really 
pays off... 


Laura’s right—templates 
make HTML much easier 
to maintain than Bob’s 
approach. We’ll dive into 
templates on the next page. 
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reusing html pages 


Temdafes U> _ 

C]«se 

Template engines let programmers apply the object-oriented notions of inheritance and reuse to 
the production of textual data, such as web pages. 

A website’s look and feel can be defined in a top-level HTML template, known as the base 
template, which is then inherited from by other HTML pages. If you make a change to the 
base template, the change is then reflected in all the HTML pages that inherit from it. 

The template engine shipped with Flask is called Jinja2, and it is both easy to use and powerful. 
It is not this book’s intention to teach you all you need to know about Jinja2, so what appears 
on these two pages is—by necessity—both brief and to the point. For more details on what’s 
possible with Jinja2, see: 

http://jinja.pocoo. org/docs/dev/ 


Here 5 s the base template we’ll use for our webapp. In this file, called base . html, we put the 
HTML markup that we want all of our web pages to share. We also use some Jinja2-specific 
markup to indicate content that will be supplied when HTML pages inheriting from this one 
are rendered (i.e., prepared prior to delivery to a waiting web browser). Note that markup 
appearing between { { and } }, as well as markup enclosed between { % and % }, is meant for the 
Jinja2 template engine: weVe highlighted these cases to make them easy to spot: 



sta^dav^d 

HTML2? 

w\av"kuf- 


<!doctype html> 
<html> 

<head> 

<title>{{ 



This is a JmjaZ directi ve, y,hidh 
i^didates that a value will be provided 
prior to «re„de<ri» 9 (thihk of this as a, 
argument to the template). 

} }</title> 


<link rel="stylesheet” href=”static/hf.css” /> 
</head> 

This 

<body> 

{% block body %} 


This stylesheet delines 
the look a*d -feel o-f all 
the web pages. 


{% endblock %} 


</body> 


^ These JmVaZ divettives aditate that 

a blodk ot rtTML vili be substituted 

heve pviov to vendering, a*d is to be 
j-j Lvi J-knF '.wkrvits 




</html> 


This is the 
tase template. 



With the base template ready, we can inherit from it usingjinja2 , s extends directive. When 
we do, the HTML files that inherit need only provide the HTML for any named blocks in the 
base. In our case, we have only one named block: body. 
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Here’s the markup for the first of our pages, which we are calling entry. html. This is markup for 
a HTML form that users can interact with in order to provide the value for phrase and letters 
expected by our webapp. 

Note how the “boilerplate” HTML in the base template is not repeated in this file, as the extends 
directive includes this markup for us. All we need to do is provide the HTML that is specific to this 
file, and we do this by providing the markup within the Jinja2 block called body: 


{% extends 'base.html' 


{% block body %} 

<h2>{{ the title }}</h2> 



This -template 'mherits 
-(Vom the base, and 
provides a replatement ■W 


fhe block talled "body 



<form method='POST' action='/search4'> 

<table> 

<p>Use this form to submit a search request:</p> 

<tr><td>Phrase : < / tdxtdxinput name=' phrase ' type='TEXT' width=' 60 ' ></tdx/tr> 
<trXtd>Letters : < /tdxtdxinput name=' letters ' type='TEXT' value= ' aeiou ' x/tdx /tr> 
</table> 


<p>When you're ready, click this button:</p> 
<p><input value='Do it!' type='SUBMIT'></p> 
</form> 


{% endblock %} 


And, finally, here’s the markup for the results . html file, which is used to render the results of 
our search. This template inherits from the base template, too: 


{% extends 'base.html' 

{% block body %} 

<h2>{{ the title }}</h2> 


As with "entry.html", this 
template also inherits 
-rv-om th e base, and also 



<p>You submitted the following data:</p> 

<table> 

<trXtd>Phrase : </tdxtd>{ { the phrase }}</tdx/tr> 
<tr><td>Letters :</tdxtd>{ { the_letters } }</tdx/tr> 
</table> 


provides a replaeemea 4r 
the block called "body". 


<p>When "{{the phrase }}" is search for 
results are returned:</p> 

<h3>{{ the results }}</h3> 


{% endblock %} 


{{ the_letters }}", the following 

Note these additional ar^meh values, wh«ch 
you need to provide values tor prior to rendering. 
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it’s just html 


Templates Relate to Web Pages 


Our webapp needs to render two web pages, and now we have two templates that can 
help with this. Both templates inherit from the base template and thus inherit the base 
template’s look and feel. Now all we need to do is render the pages. 

Before we see how Flask (together with Jinja2) renders, let’s take another look at 
our “napkin specifications” alongside our template markup. Note how the HTML 
enclosed within the Jinja2 {% block %} directive closely matches the hand-drawn 
specifications. The main omission is each page 5 s title, which we’ll provide in place of 
the { { the_title } } directive during rendering. Think of each name enclosed in 
double curly braces as an argument to the template: 


Download these templates 
(and the CSS) from here: 

http://python.itcarlow. ie/ed2/. 


We lto»< «***&*** °" WA! 




PWrase ; 

Leltev* I aetow 


VVVicm 


ve ve 


adv, tWeV tWis « 




{% extends 'base.html' %} 

{% block body %} 

<h2>{{ the_title }}</h2> 

<form method='POST' action='/search4'> 

<table> 

<p>Use this form to submit a search request:</p> 
<tr><td>Phrase : < /tdxtdxinput name=' phrase ' type='TEXT' 
width='60'></td></tr> 

<tr><td>Letters:</tdxtdxinput name= ' letters ' type='TEXT' 
value='aeiou'></td></tr> 

</table> 

<p>When you're ready, click this button:</p> 

<p><input value='Do it!' type='SUBMIT'></p> 

</form> 

{% endblock %} 
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Renderiwg Templates frow Flask 

Flask comes with a function called render_template, which, when 
provided with the name of a template and any required arguments, returns a 
string of HTML when invoked. To use render_template, add its name 
to the list of imports from the flask module (at the top of your code), then 
invoke the function as needed. 

Before doing so, however, let 5 s rename the file containing our webapp’s code 
(currently called hello_f lask. py) to something more appropriate. You 
can use any name you wish for your webapp, but we 5 re renaming our file 
vsearch4web . py. Here’s the code currently in this file: 

from flask import Flask 
from vsearch import search41etters 

app = Flask (_name_) 

@app. route (' / ') 
def hello() -> str: 

return 'Helio world from Flask!' 

@app.route('/search4') 
def do_search() -> str: 

return str(search41etters('life, the universe, and everything', 'eiru,!’)) 


This Code how \rcsidcs 
i* a -file called 
vsedv-dh^web.py”. 



app.run() 


To render the HTML form in the entry. html template, we need to make 
a number of changes to the above code: 


o 

o 


Import the render_template function 

Add render_template to the import list on the from flask line at the top of the code. 

Create a new URL—in this case , /entry 

Every time you need a new URL in your Flask webapp, you need to add a new @app .route line, 
too. We’ll do this before the app . run () line of code. 



Create a function that returns the correctly rendered HTML 

With the @app . route line written, you can associate code with it by creating a function that does 
the actual work (and makes your webapp more useful to your users). The function calls (and returns 
the output from) the render_template function, passing in the name of the template file 
(entry. html in this case), as well as any argument values that are required by the template (in the 
case, we need a value for the_title). 


Let’s make these changes to our existing code. 
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render html templates 


Pisplaying the Webapps HTML Form 

Let’s add the code to enable the three changes detailed at the bottom of the 
last page. Follow along by making the same changes to your code: 


o Irnport the render_template function 

from flask import Flask, render_template 

o Create a new URL—in this case, /entry 


Add V«dor_template" th e 

list ot tedhuologies imported 
+(ror« -the “tlask" module. 


@app. route (' /entry') 


Underneath the u do__seardh w tundtion, but 
betore the w app-runO w Ime, insert ibis Ime to 
add a new URL to the webapp- 

e Create a function that returns the correctly rendered HTML 

r Provide "the name o-f 
the temptate to render. 

return render_template (' entry. html' , 

the title='Welcome to search41etters on the web!) 



Add this -Punetion 
dired-tly uhderneath the 
hew ^PP-route" line- 



Provide a value to 
assotiate wth "the 

"-the_-Wk’ ar^ument- 


With these changes made, the code to our webapp—with the additions 
highlighted—now looks like this: 


from flask import Flask, render_template 
from vsearch import search41etters 


app = Flask (_name ) 

@app.route( 1 /') 
def hello() -> str: 

return 'Helio world from Flask!' 

@app.route('/search4') 
def do_search() -> str: 

return str(search41etters('life, the universe, and everything', 'eiru,!')) 

@app.route('/entry') 

def entry_page() -> 'html': 

return render_template('entry.html', 

the title='Welcome to search41etters on the web!') 



ror 


app.run () 
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Preparmg to Run the Template Code 

It’s tempting to open a command prompt, then run the latest version of our 
code. However, for a number of reasons, this won’t immediately work. 

For starters, the base template refers to a stylesheet called hf. css, and this 
needs to exist in a folder called static (which is relative to the folder that 
contains your code). Here’s a snippet of the base template that shows this: 


If you haverft done so 
already, download the 
templates and the CSS from 
here: 

http://python.itcarlow. ie/ed2/. 



(see the URL at the side of this page). Just be sure to put the downloaded 
stylesheet in a folder called static. 

In addition to this, Flask requires that your templates be stored in a folder 
called templates, which—like static—needs to be relative to the folder 
that contains your code. The download for this chapter also contains all three 
templates...so you can avoid typing in all that HTML! 

Assuming that you Ve put your webapp’s code file in a folder called webapp, 
here’s the structure you should have in place prior to attempting to run the 
most recent version of vsearch4web . py: 


This -foldev* £oh‘ fca 
dll of ouv webapp^ 
-files. 




<r 


vsearch4web.py 


static 


This -file ^oividihs ou\r v/ebapp^s 
£ode (showh at the bottor» of 
the last page). 



Heres -tHe stylesheet (m 
its vev-y ow> -folder)' 


hf.css 


All of owr apflitation s 
templates are stored here- 


"I 


templates 
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run that webapp 


We 're Ready for a Test Ruw 

If you have everything ready—the stylesheet and templates downloaded, and 
the code updated—you’re now ready to take your Flask webapp for another 
spin. 

The previous version of your code is likely stili running at your command 
prompt. 

Return to that window now and press Ctrl and C together to stop the previous 
webapp’s execution. Then press the up arrow key to recall the last command 
line, edit the name of the file to run, and then press Enter. Your new version 
of your code should now run, displaying the usual status messages: 


* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) 
127.0.0.1 - - [23/Nov/2015 21:51:38] "GET / HTTP/1.1" 200 - 
127.0.0.1 - - [23/Nov/2015 21:51:48] "GET /search4 HTTP/1.1" 200 - 


£>{x>Y “the 
webapy 


StM up your new todt (whidh is 

$ python3 vsearch4web. py ,h vsea\r£h4"webpy” -file). 

* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) 

Tbe new tode is up 
and vunnincji and 
waitmj h> se\rvide 
ve«\uests. 


Recall that this new version of our code stili supports the / and /search4 
URLs, so if you use a browser to request those, the responses will be the same 
as shown earlier in this chapter. However, if you use this URL: 


http://127.0.0.1:5000/entry 


the response displayed in your browser should be the rendered HTML form 
(shown at the top of the next page). The command-prompt should display 
two additional status lines: one for the /entry request and another related to 
your browser’s request for the hf . ess stylesheet: 


You be 

HTAIL 


...and youv Wovisev 
also ve«\uests the 
stylesheet- 


127.0.0.1 - - [23/Nov/2015 21:55:59] "GET /entry HTTP/1.1" 200 - 
127.0.0.1 - - [23/Nov/2015 21:55:59] "GET /static/hf.ess HTTP/1.1" 304 - 
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'esT DriVq 


Here’s what appears on screen when we type http: //127.0.0.1:5000/entry into our 
browser: 


< > EO 


127.0.0.1 


Welcome to search4letters on the web! 

Use this form to submit a search request: 


Phrase: 

Letters: 


aeiou 


When you're ready, click this button: 


Do it! 


fi & no 




We aren’t going to win any web design awards for this page, but it looks OK, and resembles what we 
had on the back of our napkin. Unfortunately, when you type in a phrase and (optionally) adjust the 
Letters value to suit, clicking the Do it! button produces this error page: 



This is a bit of a bummer, isn’t it? Let’s see what’s going on. 
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what went wrong? 


Uwderstawdmg HTTP Status Codes 

When something goes wrong with your webapp, the web server responds 
with a HTTP status code (which it sends to your browser). HTTP is the 
Communications protocol that lets web browsers and servers communicate. 
The meaning of the status codes is well established (see the Geek Bits on the 
right). In fact, every web request generates an HTTP status code response. 

To see which status code was sent to your browser from your webapp, review 
the status messages appearing at your command prompt. Here’s what we saw: 


127.0.0.1 - - 
127.0.0.1 - - 
127.0.0.1 - - 


[23/Nov/2015 21:55:59] 
[23/Nov/2015 21:55:59] 
[23/Nov/2015 21:56:54] 


"GET /entry HTTP/1.1" 200 - 

"GET /static/hf.css HTTP/1.1" 304 - 

"POST /search4 HTTP/1.1" 405 - 


Mh-oh. Soroe-thmg has gone wo«g, 
and the sevvev has genevated a 
dlieirt-m-or status iode 



The 405 status code indicates that the client (your browser) sent a request 
using a HTTP method that this server doesn’t allow. There are a handful of 
HTTP methods, but for our purposes, you only need to be aware of two of 
them: GET and POST. 

The GET method 

Browsers typically use this method to request a resource 
from the web server, and this method is by far the most 
used. (We say “typically” here as it is possible to—rather 
confusingly—use GET to send data from your browser 
to the server, but we’re not focusing on that option here.) 

All of the URLs in our webapp currently support GET, 
which is Flask’s default HTTP method. 

The POST method 

This method allows a web browser to send data to the 
server over HTTP, and is closely associated with the 
HTML <form> tag. You can teli your Flash webapp to 
accept posted data from a browser by providing an extra 
argument on the @app . route line. 


Let’s adjust the @app . route line paired with our webapp 5 s /search4 
URL to accept posted data. To do this, return to your editor and edit the 
vsearch4web . py file once more. 



Here's a quick and dirty 
explanation of the various HTTP 
status codes that can be sent 
from a web server (e.g., your Flask 
webapp) to a web client (e.g., 
your web browser). 

There are five main categories 
of status code: 10Os, 200s, 300s, 
400s, and 500s. 

Codes in the 100-199 range are 
informational messages: all is 
OK, and the server is providing 
details related to the clienfs 
request. 

Codes in the 200-299 range are 
success messages: the server 
has received, understood, and 
processed the clienfs request. All 
is good. 

Codes in the 300-399 range 
are redirection messages: the 
server is informing the client 
that the request can be handled 
elsewhere. 

Codes in the 400-499 range are 
client error messages: the server 
received a request from the client 
that it does not understand and 
can't process. Typically, the client 
is at fault here. 

Codes in the 500-599 range are 
server error messages: the server 
received a request from the client, 
but the server failed while trying 
to process it. Typically, the server 
is at fault here. 

For more details, please see: 
h ttps://en. wikipedia. org/wiki/ 
List_of_HTTP_status_codes. 
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Handling Posted Pata 


As well as accepting the URL as its first argument, the @app . route decorator 
accepts other, optional arguments. 


One of these is the methods argument, which lists the HTTP method(s) that 
the URL supports. By default, Flash supports GET for all URLs. However, if the 
methods argument is assigned a list of HTTP methods to support, this default 
behavior is overridden. Here’s what the @app . route line currently looks like: 

@app.route('/search4' 

To have the /search4 URL support POST, add the methods argument to the 
decorator and assign the list of HTTP methods you want the URL to support. 
This line of code, below, States that the /search4 URL now only supports the 
POST method (meaning GET requests are no longer supported): 



tVc have ho‘t 

spe£i-fied ah 
WTTP method 

io support here, 
so Ptask de-faults 
to 6jBT. 


@app.route('/search4', methods=['POST']) 



This small change is enough to rid your webapp of the “Method Not Allowed” 
message, as the POST associated with the HTML form matehes up with the 
POST on the @app . route line: 


The VseavM” 
URL how 

supports ohly the 
?0ST method- 




What if I need my URL to support both the GET method as well as POST? Is that possible? 


Yes, all you need to do is add the name of the HTTP method you need to support to the list assigned to the methods arguments. For 
example, if you wanted to add GET support to the /search4 URL, you need only change the @app. route line of code to look like this: @ 
app . route (' /search4 ', methods= [ ' GET ' , ' POST ' ] ). For more on this, see the Flask docs, which are available 

here http://flask.pocoo.org. 
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switch on debugging 


Ref iwmg the Edit/Stop/Start/Tcst Cycle 

At this point, having saved our amended code, it’s a reasonable course of action to stop 
the webapp at the command prompt, then restart it to test our new code. This edit/ 
stop/start/test cycle works, but becomes tedious after a while (especially if you end up 
making a long series of small changes to your webapp’s code). 

To improve the efficiency of this process, Flask allows you to run your webapp in 
debugging mode , which, among other things, automatically restarts your webapp every 
time Flask notices your code has changed (typically as a resuit of you making and 
saving a change). This is worth doing, so let’s switch on debugging by changing the last 
line of code in vsearch4web . py to look like this: 

app. run (debug=True) -Swi-fcdhes debugging 

Your program code should now look like this: 

from flask import Flask, render_template 
from vsearch import search41etters 

app = Flask (_name_) 

@app. route (' / ') 
def hello() -> str: 

return 'Helio world from Flask!' 

@app.route('/search4', methods=['POST']) 
def do_search() -> str: 

return str(search41etters('life, the universe, and everything', 'eiru,!')) 

@app.route('/entry') 

def entry_jpage () -> 'html': 

return render_template('entry.html', 

the title='Welcome to search41etters on the web!') 


app.run(debug=True) 


We are now ready to take this code for a test run. To do so, stop your currently 
running webapp (for the last time) by pressing Ctrl-C , then restart it at your command 
prompt by pressing the up arrow and Enter. 


Rather than showing the usual “Running on http : / / 12 7 . . . ” message, Flask 
spits out three new status lines, which is its way of telling you debugging mode is now 
active. Here’s what we saw on our computer: 


$ python3 vsearch4web.py 

* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) 

* Restarting with stat 

* Debugger is active! 

* Debugger pin code: 228-903-465 


Now that we are up and running again, let’s interact with our webapp once more and 
see what’s changed. 


This is Flask s way o( tellmj you that 
yowr webapp will automatidally restari it 
youv tode dhanges. /\lso ; doit-fc worry i-f 
youv- debugger pi„ tode is di-fterent -from 
ouvs (that S 0\Q- I/Ve woh^t use this pm. 
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Tqst Driiq- 

Return to the entry form by typing http://12 7. 0.0.1:5000/entry into your browser: 


• • • < > | ED 127.0.0.1 C [b (3 



The “Method Not Allowed” error has gone, but things stili arerTt working right. You can type any phrase 
into this form, then click the Do it! button without the error appearing. If you try it a few times, you’11 
notice that the results returned are always the same (no matter what phrase or letters you use). Let’s 
investigate what’s going on here. 


• ••<,> EU 

127 . 0 . 0.1 

6 ] 

cb 

s , 



{'uVeWViVr'} 



ata/a^s -the sa**e. 
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where’s the data? 


Accessing HTML Forw Pata with Flask 

Our webapp no longer fails with a “Method Not Allowed” error. Instead, it 
always returns the same set of characters: u, e , commo , z, and r. If you take 
a quick look at the code that executes when the /search4 URL is posted to, 
youdl see why this is: the values for phrase and letters are hardcoded into 
the function: 



Our HTML form posts its data to the web server, but in order to do 
something with the data, we need to amend our webapp 5 s code to accept the 
data, then perform some operation on it. 


Flask comes with a built-in object called reque st that provides easy access 
to posted data. The re que st object contains a dictionary attribute called 
form that provides access to a HTML form 5 s data posted from the browser. 
As form is like any other Python dictionary, it supports the same square 
bracket notation you first saw in Ghapter 3. To access a piece of data from 
the form, put the form elemenfs name inside square brackets: 


{% extends 'base.html' %} 


The data -(Vom -this 
+01™ ele*»eht is available 
y OWr webapps code as 

Ve^uest.tov»«rphvase'J" 


{% block body %} 

<h2>{{ the_title }}</h2> 

<form method='POST' action='/search4'> 

<table> 

<p>Use this form to submit a sea^ 

<tr><td>Phrase: </tdxtdxinput Q}ame=' phras€^“ - cppe= ' TEXT ' 
width='60'></td></tr> 

<trXtd>Letters : </tdxtdxinput (name= ' letter'^ type= ' TEXT ' 
value=' aeiou' ></tdx/tr> 

</table> 

<p>When you're ready, click this button:</p> 

<p><input value='Do it!' type= ' SUBMIT ' x/p> 

</form> 


{% endblock %} 



127.0.0.1 


Welcome to search4letters on the web! 

Use this tu,.' to submit a search request: 


ady, click this button: 



The WT/V[L temptate (i* 
the w eht\ry.htmr 
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The ve*dev-ed -fov-m 
m ouv* web bvowsev 


Tke da-ta W tW.s tov» 
elegent available W» owv vjebapp 

as 'Ve<\uest toy-rr,C'le-t-teys J • 
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Using Request Pata m Your Webapp 


To use the re que st object, import it on the f rom f lask line at the top of your 
program code, then access the data from re que st. f orm as needed. For our 
purposes, we want to replace the hardcoded data value in our do_search function 
with the data from the form. Doing so ensures that every time the HTML form is 
used with different values for phrase and letters, the resuits returned from our 
webapp adjust accordingly. 


Lefs make these changes to our program code. Start by adding the reque st object 
to the list of imports from Flask. To do that, change the first line of vsearch4web . 
py to look like this: 



from flask import Flask, render_template, request 

We know from the information on the last page that we can access the phrase 
entered into the HTML form within our code as request. form [ ' phrase ’ ], 
whereas the entered letters is available to us as request. form [ ' letters ' ]. 
Lefs adjust the do_search function to use these values (and remove the hardcoded 
strings): 


-to 

the lis-t o-p 
impov-ts. 


Oea-te *bwo 

VdvidWcS*** r — 

phrase = request.form['phrase'] 


@app.route('/search4', methods=['POST']) 
def do search() -> str: 


fh assi 3 h WT/VJL da-fca 

v ^ he '"ly Cveated vaviables.. 

^ letters = request.form['letters'] j 

return str(search41etters(phrase, letters)) 



Automatic Reloads 


use “the 
vav-iablcs m tali *to 
w scav-tK / Hc'tWs w . 


Now...before you do anything else (having made the changes to your program code 
above) save your vsearch4web . py file, then flip over to your command prompt and 
take a look at the status messages produced by your webapp. Here’s what we saw (you 
should see something similar): 


$ python3 vsearch4web.py 


* Restarting with stat 


* Debugger is active! 


Flask debu00ev~ 
bas spo-fc-fced the . 
tode changes, and 
vestavted yowv . 
webapp tov you. ^ 
Preily hahdy, eh? 


* Debugger pin code: 228-903-465 

127.0.0.1 - - [23/Nov/2015 22:39:11] "GET /entry HTTP/1.1" 200 - 
127.0.0.1 - - [23/Nov/2015 22:39:11] "GET /static/hf.css HTTP/1.1" 200 - 
127.0.0.1 - - [23/Nov/2015 22:17:58] "POST /search4 HTTP/1.1" 200 - 

* Detected change in 'vsearch4web.py', reloading 

* Restarting with stat 

* Debugger is active! 

* Debugger pin code: 228-903-465 


Don’t panic if you see something other than what 5 s shown here. Automatic reloading 
only works if the code changes you make are correct. If your code has errors, the 
webapp bombs out to your command prompt. To get going again, fix your coding 
errors, then restart your webapp manually (by pressing the up arrow , then Enter). 
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works better now 



Tqst Drwq 


Now that we’ve changed our webapp to accept (and process) the data from our HTML form, we can 
throw different phrases and letters at it, and it should do the right thing: 


< > ED 


v 127.0.0.1:5000/entry 


Welcome to search4letters on the web! 


Use this form to submit a search request: 


J 


< > ED 


Phrase: This is a test of the posting capability 
Letters: aeiou 

When you're ready, click this button 

Do it! 


{'oVeViVa'} 



< > ED 


127.0.0.1:5000/entry 


Welcome to search4letters on the web! 

Use this form to submit a search request: 


Phrase: life, the universe, and everything 
Letters: xyz 


I " 

m 


< > ED 


When you're ready, click this button: 


Do it! 



< > ED 


t 127.0.0.1:5000/entry 


Welcome to search4letters on the web! 

Use this form to submit a search request: 



di & 


127.0.0.1 


The phvase £oirfcaihS ali but ohe 
0+ -the letters posted to the 
web sevvev. 


di 0 |_|_ 


127.0.0.1 


0y\ ly -the lettev y appeavs 
in -the posted phvase- 


di & [+" 


di & QT 


di 0 r+~ 


127.0.0.1 


di 


& 


+ 


Remembev: an empty set appeavs as VtO'^ 
so this means none of the letters m, n, o, 
y, ov \ appeav m the phvase- 
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Producing the Results As HTML 


At this point, the functionality associated with our webapp is working: any 
web browser can submit a phrase/letters combination, and our 
webapp invokes search41etters on our behalf, returning any results. 
However, the output produced isn’t really a HTML webpage—it 5 s just the 
raw data returned as text to the waiting browser (which displays it on screen 

Recall the back-of-the-napkin specifications from earlier in this chapter. This 
is what we were hoping to produce: 


This par-t is deme The Vlv-y hW' template 
produtes an appvo*imation ok this W tor us. 


\ 


Wtltce to «avMletW oo the Vfcb! 



tteirc are youv results: 

/ou submiiied the tollowmg data: 

Ph " aSe: hitdh-hikev 

Lettevs: aeiou 

Ki ifaUhikt rr is “«w w 

the -rollowing results are retur* e d: 

f V, V } 


j 




J 


1 ^ is pa^t remaihs to be 

. ■ done- /\t the w>o*we»t) 

L. . ~ ' we ’ lre only displaymg the 

V*CSul”ts olS V*dW (JjL 

When we learned about Jinja2 5 s template technology, we presented two 
HTML templates. The first, entry. html, is used to produce the form. The 
second, results . html, is used to display the results. Let 5 s use it now to 
take our raw data output and turn it into HTML. 

tli©r©i£ir© n° 

- Dtimb Questions - 


O: 


It is possible to use Jinja2 to template textual data other than HTML? 


Yes. Jinja2 is a text template engine that can be put to many uses. That said, its typical use case is with web development projects (as 
used here with Flask), but there’s nothing stopping you from using it with other textual data if you really want to. 
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one more template 


Calculatmg the Pata We Need 


Let’s remind ourselves of the contents of the results . html template as presented 
eadier in this chapter. The Jinja2-specific markup is highlighted: 


This is 
Vsults. 

hur. 



The highlighted names enclosed in double cudy braces are Jinja2 variables that take 
their value from corresponding variables in your Python code. There are four of these 
variables: the_title, the_phrase, the_letters, and the_results. Take 
another look at the do_search function’s code (below), which we are going to adjust 
in just a moment to render the HTML template shown above. As you can see, this 
function already contains two of the four variables we need to render the template 
(and to keep things as simple as possible, weVe used variable names in our Python 
code that are similar to those used in the Jinja2 template): 


tteve ave *two 
o£ -the -fouv 
values we need- 



@app.route('/search4', methods=['POST']) 
def do_search() -> str: 

phrase = request.form['phrase'] 

letters = request.form['letters'] 

return str(search41etters(phrase, letters)) 


The two remaining required template arguments (the_title and the_results) 
stili need to be created from variables in this function and assigned values. 

We can assign the "Here are your results : " string to the_title, and then 
assign the call to search4letters to the_results. All four variables can then 
be passed into the results . html template as arguments prior to rendering. 
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building a webapp 


T em piat e Magne ts 

The Heod First authors got together and, based on the requirements 
for the updated do_search function outlined at the bottom of the 
last page, wrote the code required. In true Head First style, they did so 
with the help of some coding magnets...and a fridge (best if you don't 
ask). Upon their success, the resulting celebrations got so rowdy that 
a certain series editor bumped into the fridge (while singing the beer 
song) and now the magnets are ali over the floor. Your job is to stick 
the magnets back in their correct locations in the code. 

from flask import Flask, render_template, request 
from vsearch import search41etters 

app = Flask( name ) 


@app.route {'/') 
def hello() -> str: 

return 'Helio world from Flask!' 

@app.route ( ’/search4’, methods=[’POST']) 

def do_search() -> : 

phrase = request.form[’phrase’] 
letters = request.form[’letters’] 


return 


Deoide whi£h £ode 

goes ih tacM 
"tta dclshcd—lihc 




@app.route('/entry’) 

def entry_page() -> ’html’: 

return render_template('entry.html’ , 

the title='Welcome to search41etters on the web!’) 


Hcvc av-e ikt 
magne-b y ou ■ 
Us ve -to 

VJltV»- 


app.run(debug=True) 
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magnets all arranged 



Temptate Magnets Solution 

Having made a note to keep a future eye on a certain series editor's 
beer consumption, you set to work restoring all of the code magnets 
for the updated do_search function. Your job was to stick the 
magnets back in their correct locations in the code. 

Here's what we came up with when we performed this task: 


from flask import Flask, render_template, request 
from vsearch import search41etters 


app = Flask( name ) 


@app.route {'/') 
def hello() -> str: 

return 'Helio world from Flask!' 


Cha^e the awo-taW to 
inditate that this WtU» 
now v-etuvns HTML, not 3 
fla'm-te*t stvin^ ^ as m 
^revioiAS vevsion ot this (.ode.) 


@app.route('/search4', methods=['POST']) 
def do search() -> 


Oeate 3 Python 
vavialole talled 

“feti et 



phrase = request.form['phrase'] 
letters = request.form['letters'] 



Here are your results:' 



3hd assigh a 

tring io "hile". 


r 

Cv-ea-te a^cthcv" 
Python vaviablc 
dalled l Vesul*fcs w *» 


Rendev the \esults.ht»l 
-temflate- Reme»»hev : this 
tem^late e*\>etts W 
avguweyrt valucs. 


str(search41etters(phrase, letters)) 



render template('results.html 


the_phrase=phrase, 
the letters=letters 


the title=title. 


the results=results 



...and assi yr\ 

-the v-esul-ts 

ok the tali to ^ 

"seavth^letters” 

is 'Vesults • 


t°°<Ld ilosi,,, 9 P^^thesis 

@app. route ( ' /entry') & ™ ******* ^ 

def entry_page() -> 'html': 

return render_template('entry.html', 

the title='Welcome to search41etters on the web!') 


Ea^h Pytho* vaHable 

is assigned "to rts 

^oirircspohdihg JinjaZ 

/h -this 

da-fca 4Vor* 

oum pirogiram Codt 
is passcd ih-fco -the 
tc^pla-tc. 


app.run(debug=True) 

Now that the magnets are back in their correct locations, make these code 
changes to your copy of vsearch4web . py. Be sure to save your file to ensure 
that Flask automatically reloads your webapp. We’re now ready for another test. 
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building a webapp 



Let’s test the new version of our webapp using the same examples from earlier in this chapter. Note 
that Flask restarted your webapp the moment you saved your code. 


< > E 


V 127.0.0.1:5000/entry 


di & 



Welcome to search4letters on th 

Use this form to submit a search request: 



E 


127.0.0.1 


di 0 [+ 


Phrase: This is a test of the posting capability 
Letters: aeiou 


Here are your results: 

You submitted the following data: 


When you're ready, click this button 


Do it! 


T 


< > E 


Welcome to search4letters 

Use this form to submit a search request: 

Phrase: life, the universe, and everything 
Letters: xyz 

When you're ready, click this button: 



Phrase: This is a test of the posting capability 
Letters: aeiou 


When "This is a test of the posting capability" is search for "aeiou", the following 
results are returned: 

{'e', 'i', 'a', 'o'} 


V. 127.0.0.1:5000/entry 


127.0.0.1 


Here are your results: 

You submitted the following data: 



Phrase: 

life, the universe, and everything 

Letters: 

xyz 


When "life, the universe, and everything" is search for "xyz", the following results are 
returned: 

{'y'} 


E 


127.0.0.1 


Welcome to search4le 

Use this form to submit a search requ 

_ _■! 


Phrase: hitch-hiker 
Letters: mnopq 


Here are your results: 

You submitted the following data: 


When you're ready, click this button: 


Do it! 



Phrase: 

hitch-hiker 

Letters: 

mnopq 


When "hitch-hiker" is search for "mnopq", the following results are returned: 

set() 


We re looking good -fov 

ih P u * »ow. 


I 



di s 1 In¬ 


di & nr 
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a litti e redirection 


Addiwg a Fmishiwg Touch 

Let’s take another look at the code that currently makes up vsearch4web . py. 
Hopefully, by now, all this code should make sense to you. One small syntactical 
element that often confuses programmers moving to Python is the inclusion of 
the final comma in the call to render_template, as most programmers feel 
this should be a syntax error and shouldnl be allowed. Although it does look 
somewhat strange (at first), Python allows it—but does not require it—so we can 
safely move on and not worry about it: 


from flask import Flask, render_template, request 
from vsearch import search41etters 

app = Flask (_name_) 

@app.route ('/ ') 
def hello() -> str: 

return 'Helio world from Flask!' 


@app.route('/search4', methods=['POST']) 
def do_search() -> 'html': 

phrase = request.form['phrase'] 

letters = request.form['letters'] 

title = 'Here are your results:' 

results = str(search41etters(phrase, letters)) 

return render_template('results.html', 

the_title=title, 
the_phrase=phrase, 
the_letters=letters, 
the results=results,) 


@app.route('/entry') 

def entry_page() -> 'html': 

return render_template('entry.html', 

the title='Welcome 


This ex-tira Comma looks a 

Ijttle stranje, bu-t is pertec-tly 

+,he (thougb op-tiohal) Py-th 
syivtax. ' 


Oh 


to search41etters on the web!') 


app.run(debug=True) 


This version of our webapp supports three URLs: /, / 'search4 , and / 'entry , with 
some dating back to the very first Flask webapp we created (right at the start of 
this chapter). At the moment, the / URL displays the friendly, but somewhat 
unhelpful, “Helio world from Flask!” message. 

We could remove this URL and its associated hello function from our code (as 
we no longer need either), but doing so would resuit in a 404 “Not Found” error 
in any web browser contacting our webapp on the / URL, which is the default 
URL for most webapps and websites. To avoid this annoying error message, let’s 
ask Flask to redirect any request for the / URL to the /entry URL. We do this by 
adjusting the hello function to return a HTML redirect to any web browser 
that requests the / URL, effectively substituting the /entry URL for any request 
made for /. 
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Redirect to Avoid Uwwawted Errore 


To use FlasUs redirection technology, add redirect to the f rom f lask 
import line (at the top of your code), then change the hello function’s code to 
look like this: 


from flask import Flask, render_template , request, redirect 
from vsearch import search41etters 



Add 

Vedi reti* 

"to ihe Ii st 
of impoy-ts. 


app = Flask (_name_) 

@app.route ('/') 
def hello() -> '302': 

return redirect('/entry') 


Adjust ihe anno-btion -to r«ove cleav-ly indicate whats 
be.ng vetwrned by this tundW Redall that HTTP 
status codes in the 300-3fl ran 9 c a,e ,edi,ections, 

11 ^A- l» ' S W ^ ^ 3S k SCh< ^ S y°^ b\rowse\r u/hen 

'rtdirtti is invoked. 


The v-est ofe 
-the dode — 
vc^ains 
undhanged- 



Call Flask's Vedivect" Uti» 

to instviACt the bvowsev to 

y-e^est an altevnatWe URL Cw 
this case, Ve»tvy' )• 


This small edit ensures our webapp’s users are shown the HTML form should 
they request the /entry or / URL. 

Make this change, save your code (which triggers an automatic reload), and then 
try pointing your browser to each of the URLs. The HTML form should appear 
each time. Take a look at the status messages being displayed by your webapp at 
your command prompt. You may well see something like this: 


r 


You saved youv 
dode, so Flask 
v-eloaded youv- 
webapp- 


Detected change in 'vsearch4web.py', reloading 
Restarting with stat 
Debugger is active! 

Debugger pin code: 228-903-465 


A 'request is made ^*o\r the 
Verrbry" URL, and ii is 
sewed up immediately. No-te 
ihe Z00 status dode (and 
\remembe\r -(Vom ea^liev in 
this dhapteir that dodes in 
the ZOO—Z*}*} vange av-e 
suddess messages- the sev-vev* 
has redeived, undersiood, and 
pv-odessed the diient s request). 


127 

127 

127 


0 

0 

0 


0 

0 

0 


1 

1 

1 


[24/Nov/2015 16:54:13] 
[24/Nov/2015 16:56:43] 
[24/Nov/2015 16:56:44] 


M GET 

M GET 


ff 


GET 


/entry HTTP/1.1’ 
/ HTTP/1.1" 302 
/entry HTTP/1.1' 


200 - 


200 


When a ve<\uesi is made -fov the / URL> ouv- webapp 
-f ivst vesponds with the lOZ \redi\redtion, and then the 
web bvowsev- sends anothev re<\uest (ov the 7entvy URL, 
v/hidh is suddess-Cully sewed up by ouv webapp (agam, note 
the 2-00 status dode). 

As a strategy, our use of redirection here works, but it is somewhat wasteful—a 
single request for the / URL turns into two requests every time (although client- 
side caching can help, this is stili not optimal). If only Flask could somehow 
associate more than one URL with a given function, effectively removing the need 
for the redirection altogether. That would be nice, wouldn’t it? 
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no more redirection 


Functions Can Have Multiple URLs 

It’s not hard to guess where we are going with this, is it? 

It turns out that Flask can indeed associate more than one URL with a given function, 
which can reduce the need for redirections like the one demonstrated on the last page. 
When a function has more than one URL associated with it, Flask tries to match each 
of the URLs in turn, and if it finds a match, the function is executed. 

It’s not hard to take advantage of this Flask feature. To begin, remove redirect 
from the f rom flask import line at the top of your program code; we no longer 
need it, so let 5 s not import code we don’t intend to use. Next, using your editor, cut 
the @app . route ( ' / f ) line of code and then paste it above the @app . route ( ' / 
entry ’ ) line near the bottom of your file. Finally, delete the two lines of code that 
make up the hello function, as our webapp no longer needs them. 

When you’re done making these changes, your program code should look like this: 



The "hello* 

-funC-Lon 
has been 
v-ewxoved- 


from flask import Flask, render_template, request 
from vsearch import search41etters 

app = Flask (_name_) 

@app.route('/search4', methods=['POST']) 
def do_search() -> 'html': 

phrase = request.form['phrase'] 

letters = request.form['letters'] 

title = 'Here are your results:' 

results = str(search41etters(phrase, letters)) 

return render_template('results.html', 

the_title=title, 
the_phrase=phrase, 
the_letters=letters, 
the results=results,) 


VVe y\o lon^ev- need “to tr*\>ort 
Vediv-ecV\ so we ve vemoved it 
-(*vow\ “this iirv\^oV“t line- 


@app. route (* /') ^ The eft^lry |^3^e now 

@app. route (' /entry') has -two URLs associated with it- 

def entry_j?age () -> 'html': 

return render_template('entry.html', 

the title='Welcome to search41etters on the web!') 


app.run(debug=True) 


Saving this code (which triggers a reload) allows us to test this new functionality. If 
you visit the / URL, the HTML form appears. A quick look at your webapp’s status 
messages confirms that processing / now results in one request, as opposed to two (as 
was previously the case): 

* Detected change in 'vsearch4web.py' 

* Restarting with stat 

* Debugger is active! 

* Debugger pin code: 228-903-465 
127.0.0.1 - - [24/Nov/2015 16:59:10] " 



As always, -the new version 
°+ owr webapp veloads. 

One v-e<\uest, one 

vesponsc- Tbat ^ wov-c 

like it © 
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Updatiwg What We Kwow 

WeVe just spent the last 40 pages creating a small webapp that exposes the 
functionality provided by our search41etters function to the Wodd Wide 
Web (via a simple two-page website). At the moment, the webapp runs locally on 
your computer. In a bit, we’ll discuss deploying your webapp to the cloud, but for 
now let 5 s update what you know: 



BULLET POINTS 


■ You learned about the Python Package Index 
(PyPI), which is a centralized repository for 
third-party Python modules. When connected 
to the Internet, you can automatically install 
packages from PyPI using pip. 

■ You used pip to install the Flask micro-web 
framework, which you then used to build your 
webapp. 

■ The_ _name _value (maintained by the 

interpreter) identifies the currently active 
namespace (more on this later). 


The @ Symbol before a function’s name 
identifies it as a decorator. Decorators let you 
change the behavior of an existing function 
without having to change the function’s code. In 
your webapp, you used Flask’s @app. route 
decorator to associate URLs with Python 
functions. A function can be decorated more 
than once (as you saw with the do_search 
function). 

You learned how to use the Jinja2 text template 
engine to render HTML pages from within your 
webapp. 


Is that all there Is to this chapter? 

You’d be forgiven for thinking this chapter doesn’t introduce much new Python. It 
doesn’t. However, one of the points of this chapter was to show you just how few 
lines of Python code you need to produce something that’s generally useful on 
the Web, thanks in no small part to our use of Flask. Using a template technology 
helps a lot, too, as it allows you to keep your Python code (your webapp 5 s logic) 
separate from your HTML pages (your webapp’s user interface). 

It’s not an awful lot of work to extend this webapp to do more. In fact, you could 
have an HTML whiz-kid produce more pages for you while you concentrate on 
writing the Python code that ties everything together. As your webapp scales, this 
separation of duties really starts to pay off. You get to concentrate on the Python 
code (as you’re the programmer on the project), whereas the HTML whiz-kid 
concentrates on the markup (as that’s their bailiwick). Of course, you both have to 
learn a little bit about Jinja2 templates, but that 5 s not too difficult, is it? 
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gotta love pythonanywhere 


Prepariwg Your Webapp for the Cloud 


With your webapp working to specification locally on your computer, it 5 s 
time to think about deploying it for use by a wider audience. There are lots 
of options here, with many different web-based hosting setups available to 
you as a Python programmer. One popular Service is cloud-based, hosted on 
AWS, and is called PythonAnywhere. We love it over at Head First Labs. 

Like nearly every other cloud-hosted deployment solution, PythonAnywhere 
likes to control how your webapp starts. For you, this means PythonAnywhere 
assumes responsibility for calling app . run () on your behalf, which means 
you no longer need to call app . run () in your code. In fact, if you try to 
execute that line of code, PythonAnywhere simply refuses to run your webapp. 

A simple solution to this problem would be to remove that last line of code 
from your file before deploying to the cloud. This certainly works, but means 
you need to put that line of code back in again whenever you run your 
webapp locally. If you’re writing and testing new code, you should do so 
locally (not on PythonAnywhere ), as you use the cloud for deployment only, 
not for development. Also, removing the offending line of code effectively 
amounts to you having to maintain two versions of the same webapp, one 
with and one without that line of code. This is never a good idea (and gets 
harder to manage as you make more changes). 


It would be nice if there were a way to selectively execute code based on 
whether you’re running your webapp locally on your computer or remotely 
on PythonAnywhere... 


0 



I've looked at an awful lot of Python 
programs oniine, and many of them 
contain a suite near the bottom that starts 
with: if 


name 


main 


Would something like that help here? 


Yes, that’s a great suggestion. 

That particular line of code is used in lots of 
Python programs. It 5 s affectionately referred to as 
‘dunder name dunder main.” To understand why 
it 5 s so useful (and why we can take advantage of 
it with PythonAnywhere ), let 5 s take a closer look at 
what it does, and how it works. 
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Dunder jv[atne funder Ataln Up C]«se - 

To understand the programming construet suggested at the bottom of the last page, let’s look at 
a small program that uses it, called dunder . py. This three-line program begins by displaying 

a message on screen that prints the currently active namespace, stored in the_name_ 

variable. An if statement then checks to see whether the value of name_is set to 

main_, and—if it is—another message is displayed confirming the value of_name (i.e., 

the code associated with the if suite exeeutes): 


The U dunder.p/ 
^>vo(yr3w\ Lo&t 
-thvee Imes o£ ii- 



print('We start off in:', 

if _name_ == '_main_' 

print('And end up in: 


name ) 



name ) 


Displays -the value 
" r>ame " 


Use your editor (or IDLE) to create the dunder . py file, 
then run the program at a command prompt to see what 
happens. If you’re on Windows , use this command: 

C:\> py -3 dunder.py 



vndi» 


If you are on Linux or Mac OS X, use this command: 

$ python3 dunder.py 


No matter which operating system you’re running, the dunder . py program—when exeeuted 
directly by Python—produces this output on screen: 


We start off in: mam 


And end up in: 


lUGL^ll 

main 


■} 


So far, so good. 


Me* exe£u-fced 
diretily by Pyth 

Oh, 

bo-th Calis io "prini" 
display output 


Now, look what happens when we import the dunder . py file (which, remember, is also a 
module) into the >>> prompt. We’re showing the output on Linux/Mac CUXhere. To do the 
same thing on Windows , replace python3 (below) with py -3: 


$ python3 

Python 3.5.1 ... 

Type "help", "Copyright" 

>>> import dunder 

We start off in: dunder 


, "credits" or "license" for more information. 

<$■_Look at tw.s: Ws omly a Ime displayed (as 

io W>, as" — set 


Here’s the bit you need to understand: if your program code is exeeuted directly by Python, an 

if statement like the one in dunder . py returns True, as the active namespace is_ main_. 

If, however, your program code is imported as a module (as in the Python Shell prompt example 

above), the if statement always returns False, as the value of_name_is not_main_, 

but the name of the imported module (dunder in this case). 
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dunder or wonder 


Exploitmg Punder Name Punder Main 

Now that you know what dunder name dunder main does, let 5 s exploit it to solve the 
problem we have with PythonAnywhere wanting to execute app . run () on our 
behalf. 

It turns out that when PythonAnywhere executes our webapp code, it does so by 
importing the file that contains our code, treating it like any other module. If 
the import is successful, PythonAnywhere then calls app . run () . This explains 
why leaving app . run () at the bottom of our code is such a problem for 
PythonAnywhere , as it assumes the app . run ( ) call has not been made, and fails to 
start our webapp when the app .run ( ) call has been made. 

To get around this problem, wrap the app .run () call in a dunder name dunder 
main if statement (which ensures app . run () is never executed when the 
webapp code is imported). 

Edit vsearch4web . py one last time (in this chapter, anyway) and change the 
final line of code to this: 


if 


name == '_main_ 

app.run(debug=True) 



This small change lets you continue to execute your webapp locally (where the 
app . run () line will execute) as well as deploy your webapp to PythonAnywhere 
(where the app . run () line won’t execute). No matter where your webapp runs, 
you Ve now got one version of your code that does the right thing. 


The “app.vuhO" |i he 

Ot Lodt how ohly 
rUKS whch 

divefi-tly by pyth oh. 


Peploying to PythonAnywhere (well... almost) 

All that remains is for you to perform that actual deployment to PythonAnywhere 3 s 
cloud-hosted environment. 

Note that, for the purposes of this book, deploying your webapp to the cloud 
is not an absolute requirement. Despite the fact that we intend to extend 
vsearch4web.py with additional functionali ty in the next chapter, you do 
not need to deploy to PythonAnywhere to follow along. You can happily continue 
to edit/run/test your webapp locally as we extend it in the next chapter (and 
beyond). 

However, if you really do want to deploy to the cloud, see Appendix B , which 
provides step-by-step instructions on how to complete the deployment on 
PythonAnywhere. It 5 s not hard, and won’t take more than 10 minutes. 

Whether you're deploying to the cloud or not, we’ll see you in the next chapter, 
where we’ll start to look at some of the options available for saving data from 
within your Python programs. 
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Chapter 5'$ Code 


from flask import Flask 

from vsearch import search41etters 

app = Flask( name ) 


@app.route {'/') 
def hello() -> str: 

return 'Helio world from Flask!' 

@app.route('/search4' ) 
def do_search() -> str: 

return str(search41etters('life, the universe, and everything', 'eiru,!')) 
app.run () 


/ 

Thjs is VlloJW. 

py , ouv -fiv-st webapp 
based on Flask (o»e o-f 
Python s midvo—web 
Famewoirk tedhuologies). 


Tbis is u vseav*db^t"web.py w . 

Tbis webapp e*posed tbe 
'fuyvdtio^ali-ty pvovided by ouv* 

u seav*db^"letlev*s w -fund-bion to -^ 

tbe Wovld Wide Web. |n additio* 
to Flask, tbis dode e*ploited 
tbe JinjaZ te apiate entyne- 


Tbis is "dundev-py”» 

belped ** s windevsts^d 'tbe vevy ^ 

ba«dy "<Wev- «a«>e dundev main 

w\edbair\iswx. \ 


print('We start off in:', _name_) 

if _name_= '_main_' : 

print('And end up in:', name ) 


from flask import Flask, render_template, request 
from vsearch import search41etters 

app = Flask( name ) 


@app.route ( '/search4', methods=['POST']) 
def do_search() -> 'html': 

phrase = request.form['phrase'] 

letters = request.form['letters'] 

title = 'Here are your results:' 

results = str (search41etters (phrase, letters)) 

return render_template('results.html', 

the_title=title, 
the_phrase=phrase, 
the_letters=letters, 
the_results=results, ) 

@app.route('/') 

@app.route('/entry') 

def entry_page() -> 'html': 

return render_template('entry.html', 

the_title='Welcome to... web!') 

if _name_ == '_main_' : 

app.run(debug=True) 


you are here ► 
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6 storing and monipulating dota 


^ Where to Put Your Data + 



Sooner or later, you’ll need to safely store your data somewhere. 

And when it comes to storing data, Python has you covered. In this chapter, you’ll learn 
about storing and retrieving data from text files, which—as storage mechanisms go—may 
feel a bit simplistic, but is nevertheless used in many problem areas. As well as storing and 
retrieving your data from files, you’ll also learn some tricks of the trade when it comes to 
manipulating data. We’re saving the “serious stuff” (storing data in a database) until the next 
chapter, but there’s plenty to keep us busy for now when working with files. 


this is a new chapter 
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Poing Sometbing with Your Webapps Pata 

At the moment, your webapp (developed in Chapter 5) accepts input from 
any web browser (in the form of a phrase and some letters), performs a 
search41etters call, and then returns any results to the waiting web browser. 
Once done, your webapp discards any data it has. 



There are a bunch of questions that we could ask of the data our webapp uses. 

For instance: How many requests have been responded to? WhaPs the most common list of 
letters? Which IP addresses are the requests comingfrom? Which browser is being used the 
most? and so on, and so forth. 

In order to begin answering these (and other) questions, we need to save the 
webapp’s data as opposed to simply throwing it away. The suggestion above makes 
perfect sense: let’s log data about each web request, then—once we have the 
logging mechanism in place—go about answering any questions we have. 
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Python Supports Open, Process, Close 


No matter the programming language, the easiest way to store data is to save it 
to a text file. Gonsequently, Python comes with built-in support for open, process, 
close. This common technique lets you open a file, process its data in some 
way (reading, writing, and/or appending data), and then close the file when 
you’re done (which saves your changes). 

Here’s how to use Python’s open, process, close technique to open a file, process 
it by appending some short strings to it, and then close the file. As we’re only 
experimenting for now, let’s run our code at the Python >>> shell. 



To access the >>> prompt: 

• run IDLE on your 
computer; 


We start by calling open on a file called todos . txt, using append mode , as 
our plan is to add data to this file. If the call to open succeeds, the interpreter 
returns an object (known as a file stream ) which is an alias for the actual file. 
The object is assigned to a variable and given the name todos (although you 
could use whichever name you wish here); 

Ofen a 'file - "N ^--.-whieh has -this 

V 'Pilename--- 


run the python3 

command in a Linux or 
Mac OSX terminal; or 

use py -3 at a 
Windows command line. 


»> todos = open ( 1 todos . txt 1 , 1 a 1 ) 

-ahd oych the -Pile i* w ayyend-mode" 

I 'P d\\ is OX, 'open” returns 

a -Pile stream, whieh we^ve 
assig*ed to this variable- 



The todos variable lets you refer to your file in your code (other 
programming languages refer to this as a file handle). Now that the file is open, 
lets write to it using print. Note how, below, print takes an extra argument 
(file), which identifies the file stream to write to. We have three things to 
remember to do (ifs never-ending, really), so we call print three times: 


IA k print a messaje- ^ ^ ...to the -Pile stream. 

»> print ( f Put out the trash. 1 , file=todos) 

»> print ( f Feed the cat. 1 , file=todos) 

»> print ( 1 Prepare tax return. ' A file=todos) 


As we have nothing else to add to our to-do list, lets close the file by calling the 
close method, which is made available by the interpreter to every file stream: 


»> todos . close () 


We re done, so let's tidy *p a-Pter 
ouvselves by tlosmoj -tbe tile stvea». 


If you forget to call close, you could potentially lose data. Remembering to 
always call close is important. 
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read what’s written 


Reading Pata frow an Existing File 


Now that youVe added some lines of data to the todos.txt file, let’s look 
at the open, process, close code needed to read the saved data from the file and 
display it on screen. 

Rather than opening the file in append mode, this time you are only 
interested in reading from the file. As reading is open’s default mode, you 
don’t need to provide a mode argument; the name of the file is all you need 
here. We’re not using todos as the alias for the file in this code; instead, wedl 
refer to the open file by the name tasks (as before, you can use whichever 
variable name you want to here): 


'Headingf’ is the 

H II f .. I 

open iunction s 
default mode. 


Ope* a -file •• 



...y/hieh has -this -pilename. 


»> tasks = open ( 1 todos . txt 1 ) 


l-f all is OK u open” veiurns 

a -Pile s^vear*, whidh we^ve 
assigned *to -this vaHable- 


Let 5 s now use tasks with a for loop to read each individual line from the 
file. When we do this, the for loop’s iteration variable (chore) is assigned 
the current line of data as read from the file. Each iteration assigns a line 
of data to chore. When you use a file stream with Python’s for loop, the 
interpreter is smart enough to read a line of data from the file each time the 
loop iterates. It’s also smart enough to terminate the loop when there’s no 
more data to read: 

The "tasb" 

for chore in tasks : ^- variable is 

print (chore) the -Pile 

stream. 


Think of 


>» 


• • • 


"^bove* 3s 3n 
alias -Pov *thc 
Ime in the file- 


Put out the trash. 
Feed the cat. 

File tax return. 



The output shovjs the data tv-om 
the "todos.ta" -file- Note how the 
loof ends when we vun out of lines 

-to veadf 


As you are merely reading from an already written-to file, calling close is less 
critical here than when you are writing data. But it’s always a good idea to close a 
file when it is no longer needed, so call the close method when you’re done: 

»> tasks . close () - 


WeVe done, so let's tidy up aftev 
ouvselves by tlosing the f ile stream. 
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t]i©r©i£tr© no 

Dumb QuestiQns 


q: What’s the deal with the extra newlines on output? The data in the file is three lines long, but the for loop produced six lines 
of output on my display. What gives? 

Yes, the for loop’s output does look strange, doesrft it? To understand what’s happening, consider that the print function appends 
a newline to everything it displays on screen as its default behavior. When you combine this with the fact that each line in the file ends in a 
newline character (and the newline is read in as part of the line), you end up printing two newlines: the one from the file together with the one 
from print. To instruet print not to include the second newline, change print (chore ) to print (chore, end= ' ' ) ■ 
This has the effect of suppressing print’s newline-appending behavior, so the extra newlines no longer appear on screen. 


q 

A: 


What other modes are available to me when l’m working with data in files? 

There are a few, which weVe summarized in the following Geek Bits box. (That’s a great question, BTW.) 


Gee]c Bits 


The first argument to open is the name of the file to process.The second argument is optional. It can be set to 
a number of different values, and dictates the mode the file is opened in. Modes include "reading," "writing," and 
"appending." Here are the most common mode values, where each (except for ' r') creates a new empty file if the file 
named in the first argument doesn't already exist: 



' r ' Open a file for reading.This is the default mode and, as such, is optional. When no second argument is 
provided, ' r' is assumed. It is also assumed that the file being read from already exists. 

' w' Open a file for writing. If the file already contains data, empty the file of its data before continuing. 

' a ’ Open a file for appending. Preserve the file's contents, adding any new data to the end of the file (compare 
this behavior to ' w ’ ). 

' x 1 Open a new file for writing. Fail if the file already exists (compare this behavior to ’ w 1 and to 1 a 1 ). 


By default, files open in text mode, where the file is assumed to contain lines of textual data (e.g., ASCII or UTF-8). If 
you are working with nontextual data (e.g., an image file or an MP3), you can specify binary mode by adding "b"to 
any of the modes (e.g., 1 wb 1 means "write to a binary data"). If you include"+" as part of the second argument, the file 
is opened for reading ond writing (e.g., ' x+b ' means "read from and write to a new binary file"). Refer to the Python 
docs for more details on open (including information on its other optional arguments). 


IVe looked at a bunch of Python 
projects on GitHub, and most of them 
use a "with" statement when opening files. 
What's the deal with that? 


The with statement is more convenient. 

Although using the open function together with the close 
method (with a bit of processing in the middle) works fine, most 
Python programmers shun open, process, close in favor of the 
with statement. Let’s take some time to find out why. 
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gotta love with 


A Petter Opew, Process, Close: "with" 

Before we describe why with is so popular, let’s take a look at some code that 
uses with. Here is the code we wrote (two pages ago) to read in and display the 
current contents of our todos . txt file. Note that weVe adjusted the print 
function call to suppress the extra newline on output: 


Open “the -f 

3ss\<yan<j “the 
Jfile stv-caw 
{0 a vav-iablc- 




tasks = open('todos.txt') 
for chore in tasks: 

print(chore, end='') 
tasks.close() 

t 

Closc *the -file* 


Pet-Poirr* some pv-oCcssi 


h 9* 


Let’s rewrite this code to use a with statement. These next three lines of code use 
with to perform exactly the same processing as the four lines of code (above): 


Ope* “the -f »le* 




for chore in tasks 
print(chore, end 


id='') 


Assijh 

“the -Pile 

S"tte3hr» -{jQ 

* vamiable. 

Peirtov-n, so**e pv-oCessihfl 

phich is ih e sawe tode as 

beWe). 


Notice anything missing? The call to close does not make an appearance. 
The with statement is smart enough to remember to call close onyour behalf 
whenever its suite of code ends. 


This is actually much more useful than it initially sounds, as lots of programmers 
often forget to call close when they 5 re done processing a file. This is not such a 
big deal when all you’re doing is reading from a file, but when you’re writing to 
a file, forgetting to call close can potentially cause data loss or data corruptiori. By 
relieving you of the need to remember to always call close, the with statement 
lets you concentrate on what it is you’re actually doing with the data in the open 
file. 

The "with" statement manages cowtext 

The with statement conforms to a coding convention built into Python called the 
context management protocol. We’re deferring a detailed discussion of this 
protocol until later in this book. For now, all you have to concern yourself with is 
the fact that when you use with when working with files, you can forget about 
calling close. The with statement is managing the context within which its 
suite runs, and when you use with and open together, the interpreter cleans up 
after you, calling close as and when required. 


Python supports 
"open, process, close.” 

But most Python 
programmers preler 
to use the "with” 
statement. 
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Let’s put what you now know about working with files to use. Here is the current code for your 
webapp. Give it another read before we teli you what you have to do: 


from flask import Flask, render_template, request 
from vsearch import search41etters 

app = Flask( name ) 


@app.route('/search4', methods=['POST']) 
def do_search() -> 'html': 

phrase = request.form['phrase'] 

letters = request.form['letters'] 

title = 'Here are your results:' 

results = str (search41etters(phrase, letters)) 

return render_template('results.html', 

the_title=title, 
the_phrase=phrase, 
the_letters=letters, 
the results=results,) 


TVis is -the 

"weairdh^Ueb.py" todt 
T\rorr> Chaptcir 



@app.route ( '/') 

@app.route('/entry') 

def entry_page() -> 'html': 

return render_template('entry.html', 

the title='Welcome to search41etters on the web!') 


if _name_ == '_main_ 

app.run(debug=True) 


Your job is to write a new function, called log_request, which takes two arguments: req and 
res. When invoked, the req argument is assigned the current Flask request object, while the 
res argument is assigned the results from calling search4letters. The log_request 
function’s suite should append the value of req and res (as one line) to a file called vsearch. 
log. We’ve got you started by providing the function’s def line. You are to provide the missing 
code (hint: use with): 


"tWis 
surte heve- 


def log request(req: 'flask request', res: str) -> None: 



you are here ► 
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creating the log 



§ 0 (.Vtl 0 H 


Your job was to write a new function, called log_request, which takes two arguments: req 
and res. When invoked, the req argument is assigned the current Flask request object, while 
the res argument is assigned the results from calling search4letters. The log 
request function’s suite should append the value of req and res (as one line) to a file called 
vsearch. log. We got you started—you were to provide the missing code: 


I h,s annotatio* may have thvo*» you a little- Redall that tu»dtio» 
amnotations ave meamt -to be vead by othev pvojva^evs. They ave 
dodumetvtatioin, noi exedutable todc- the Pytho» mtevpvetev always 
ig»oves them, so you da» use a»y a»»otatio» desdviptov you like. 


def log request(req: 'flask request', res: str) -> None: 


Use VrlW’ -to w 
o^ev\ u vseav"£-hlo^ 
m a\>^d wode. 


> >/i*th oper\(Wavdhlo(^ V) as I03 : 
pvirrtta^, v-es, 'file—lo^) 



Call the BIF to 

wv-ite the values o-f Veq" 
ahd Ves" to the opehed 

-Pile. 


Note the -P*le stv-eam 
is ealled u lo<^ 'm this 
eode- 


This annotatio* uses 
Pytho^s u No^e value 
to 'mditate this -functio* 
has no v-etuvr\ value. 


Invoking the logging function 

Now that the log_request function exists, when do we invoke it? 

Well, for starters, let’s add the log_request code into the vsearch4web . py 
file. You can put it anywhere in this file, but we inserted it directly above the do_ 
search function and its associated @app . route decorator. We did this because 
we’re going to invoke it from within the do search function, and putting it 
above the calling function seems like a good idea. 

We need to be sure to call log_request before the do_search function ends, 
but after the results have been returned from the call to search41etters. 

Here’s a snippet of do_search’s code showing the inserted call: 

• • • 

phrase = request.form['phrase'] 
letters = request.form['letters'] 
title = 'Here are your results:' 
results = str(search41etters (phrase, letters)) 

log_request(request, results) 

return render template('results.html', 


Call the “lo$_ve<v»est” 
hcvc- 
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A ftuick Review 

Before taking this latest version of vsearch4web.py for a spin, let 5 s check that your code 
is the same as ours. Here’s the entire file, with the latest additions highlighted: 


from flask import Flask, render_template, request 
from vsearch import search41etters 


app = Flask( name ) 


def log_request(req: 'flask_request', res: str) -> None: 

with open('vsearch.log', 'a') as log: 

print(req, res, file=log) 

@app.route('/search4', methods=['POST']) 
def do_search() -> 'html': 

phrase = request.form['phrase'] 

letters = request.form['letters'] 

title = 'Here are your results:' 

results = str(search41etters (phrase, letters)) 

log_request(request, results) 

return render_template('results.html', 

the_title=title, 
the_phrase=phrase, 
the_letters=letters, 
the results=results,) 



are “the latcs-fc 
additiohs, whidh 
arrange to log ea£h 
web request to a -file 
^alled Veav~eh.log" 


@app.route ( '/') 

@app.route('/entry') 

def entry_page() -> 'html': 

return render_template('entry.html', 

the title='Welcome to search41etters on the web!') 


if _name_ == '_main_ 

app.run(debug=True) 


Take your webapp for a spin... 

Start up this version of your webapp (if required) at a command prompt. On Windows , use 
this command: 

C:\webapps> py -3 vsearch4web.py 

While on Linux or Mac OS X, use this command: 

$ python3 vsearch4web.py 

With your webapp up and running, let’s log some data via the HTML form. 



You r»ay have hoti£ed 
tbat hohe o-f our webapps 
-fuh^fciohS dohtaih 
dommehts. This is a 
deliberate ontissioh oh our 
part fas there^s ohly so 
mu^h roor* oh these pages, 
ahd somethihg had to 
give). Hoic that ahy tode 
you dov/hload -from this 
book s support website 
always ihdudes dommehts. 


you are here ► 
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Use your web browser to submit data to it via the webapp’s HTML form. If you want to follow along 
with what we’re doing, submit three searches using the following values for phrase and letters: 

hitch-hiker with aeiou. 

life, the universe, and everything with aeiou. 
galaxy with xyz. 

Before you begin, note that the vsearch. log file does not yet exist. 


< > ED 


kj 127.0.0.1:5000 


1 & rgr 


The 

-fiv-st 

seareh 


Welcome to search4l< 

Use this form to submit a search re< 


ED 


127.0.0.1 


di & f+“ 


Phrase: i 

hitch-hiker 

Letters: i 

aeiou 


Here are your results: 

You submitted the following data: 




When you're ready, click this button 


Do it! 


Phrase: 

hitch-hiker 

Letters: 

aeiou 


When "hitch-hiker" is search for "aeiou", the following results are returned: 

{'e', V} 




< > E 


V. 127.0.0.1:5000 


di & r+ 


r 


Welcome to search4letters on 

Use this form to submit a search request: 

Phrase: life, the universe, and everything 
Letters: aeiou 

When you're ready, click this button: 

Do it! 



E 


127.0.0.1 











di & r+~ 




Here are your results: 

You submitted the following data: 

Phrase: life, the universe, and everything 
Letters: aeiou 

When "life, the universe, and everything" is search for "aeiou", the following 
results are returned: 

{'e', 'u', 'a', V} 


The sctoY\d 

seardh 
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< > (E) 


v 127.0.0.1:5000 


(!) 


□i 


+ 


The -fchivd (ay>d -fihal) 
seareh 


Welcome to search4letters on the web! 

Use this form to submit a search request: 


ED 


v 127.0.0.1:5000/search4 


Phrase: 

galaxy 

Letters: 

xyz 


When yoiTre ready, click this button: 


Here are your results: 

You submitted the following data: 


Do it! 


Phrase: 

galaxy 

Letters: 

xyz 


When "galaxy" is search for "xyz", the following results are returned: 

{'V', 'x'} 


Pata is logged (behmd the scewes) 


Each time the HTML form is used to submit data to the webapp, the log_ 
request function saves details of the web request and writes the results to the log file. 
Immediately after the first search, the vsearch . log file is created in the same folder 
as your webapp’s code: 


0ur -file 

manarer 

displays “the - 
doyrtevrb <y f 

our u wcba^" 
-foldcv- 


• • • 





webapp 



< m ss ipii 

OIOI 

O 0 

□lol 

< 


fi 

a 

m F 

Q, Search 


Back View 

Arrange 

Aetion 

Share 

Edit Tags 

OpenTermina! 

Search 


Name 





a Date Modified 

Size 

Kind 

► static 

► templates 





12 Jan 2016, 06:58 

11 Jan 2016,19:59 

— 

Folder 

Folder 

B vsearch.log 





Today, 16:40 

650 bytes 

Log file 

‘t vsearch4web.py 





Today, 16:29 

2 KB 

Python...ce File 

v 



1 of 4 selecteo 

1, 85.33 GB available 


_ 


cb g r+ 


« 


tare it is: the 

vseaveh.U" 

•file- 


It’s tempting to consider usingyour text editor to view the vsearch.log file 5 s 
contents. But where’s the fun in that? As this is a webapp, let’s provide access to the 
logged data via the webapp itself. That way, youll never have to move away from 
your web browser when interacting with your webapp’s data. Let 5 s create a new URL, 
called / 'viewlog ,, which displays the log’s contents on demand. 


you are here ► 
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one more uri 


View the log Through Your Webapp 


You’re going to add support for the /viewlog URL to your webapp. When your webapp 
receives a request for / viewlog , it should open the vsearch.log file, read in all of its 
data, and then send the data to the waiting browser. 


Most of what you need to do you already know. Start by creating a new @app . route 
line (we’re adding this code near the bottom of vsearch4web . py, just above the 
dunder name dunder main line): 


@app. route (' /viewlog') ^- 

Having decided on the URL, next weil write a function to go with it. Let’s call our 
new function view_the_log. This function won’t take any arguments, and will 
return a string to its caller; the string will be concatenation of all of the lines of data 
from the vsearch.log file. Here 5 s the function 5 s def line: 

def view_the_log() -> str: 

Now to write the functionh suite. You have to open the file for reading. This is the open 
function’s default mode, so you only need the name of the file as an argument to 
open. Let 5 s manage the context within which our file processing code executes using a 
with statement: 

with open('vsearch.log') as log: 


Wc have 

a bmand 
new URL. 


A»>d «e V>ave a b*-a«d 
nevj tund-tio*» xbifcb 
(atfccxrdin^ 't° tbe 
annolation) «W* 3 
string- 

0pen -the 
log -Pile £>r 
readihg. 


Within the with statement’s suite, we need to read all the lines from the file. Your first 
thought might be to loop through the file, reading each line as you go. However, the 
interpreter provides a read method, which, when invoked, returns the entire contents 
of the file “in one go.” Here 5 s the single line of code that does just that, creating a new 
string called contents: 

contents = log.read() - 

With the file read, the with statemenfs suite ends (closing the file), and you are now 
ready to send the data back to the waiting web browser. This is straightforward: 

return contents 

With everything put together, you now have all the code you need to 
respond to the /viewlog request; it looks like this: 


Read the entire -Pile \ 
°»e 90" and assig* i-fc {# a 

variable (whieh w tvt eallcd 

Wen-fcs'0. 


Take ‘the list <£ » 
Imes in w e.on*tcn*ts 
and vcWn “them. 


This is all -the eode 
you need -to sudori “the 
w /viewlog URL- 


@app.route('/viewlog') 
def view_the_log() -> str: 

with open('vsearch.log') as log: 

contents = log.read() 
return contents 
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With the new code added and saved, your webapp should automatically reload. You can enter some 
new searches if you like, but the ones you ran a few pages ago are already logged. Any new searches 
you perform will be appended to the log file. Let’s use the /viewlog URL to take a look at what’s been 
saved. Type http://127. O.O.l:5000/vlewlog into your browser’s address bar. 

Here’s what we saw when we used Safari on Mac OSX(we also checked Firefox and Chrome, and 
got the same output): 


QO 


127.0.0.1:5000/viewlog 


c 


fi 3 


+ 


{'i','e'} {'i','eVuVa'} {'a'} 


VVc’vc vw» tbvee seavthes si»te addm^ 

tode, and iw.s looks like iWee sets ok 
veswlis. M wkafs ka^e»ed ike ve<\west 
data? It appeavs to be ***" th ' s 

oi*t ? «*l?!? 



Wbere to start when things go wrowg with your output 

When your output doesn’t quite match what you were expecting (which is the case above), it’s best 
to start by checking exactly what data the webapp sent you. It’s important to note that what’s just 
appeared on screen is a rendering (or interpretation) of the webapp’s data as performed by your web 
browser. Alt the major browsers allow you to view the raw data received with no rendering applied. 
This is known as the source of the page, and viewing it can be a useful debugging aid, as well as a 
great first step toward understanding what’s going on here. 

If you are using Firefox or Chrome , right-click on your browser window and select View Page 
Source from the pop-up menu to see the raw data as sent by your webapp. If you are running 
Safari , youTl first need to enable the developer options: open up Safarfs preferences, then switch on 
the Show Develop menu in the menu bar option at the bottom of the Advanced tab. Once you do this, you 
can return to your browser window, right-click, and then select Show Page Source from the pop- 
up menu. Go ahead and view the raw data now, then compare it to what we got (on the next page). 
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use the source 


Exawiwe the Raw Pata with View Source 

Remember, the log_request function saves two pieces of data for each 
web request it logs: the request object as well as the results of the call to 
search41etters. But when you view the log (with / viewlog ), you’re only seeing 
the results data. Does viewing the source (i.e., the raw data returned from the 
webapp) offer any clue as to what happened to the request object? 

Here’s what we saw when we used Firefox to view the raw data. The fact that each 
request objecFs output is colored red is another clue that something is amiss with 
our log data: 



http://127.0.0.1:5000/viewlog 


/ - 

http://127.0.0.1:5000/viewlog 



view-source:http://127.0.0.1:5000/viewlog C ^ Search 


☆ S C * ft © 


0j RTE News - 


<Request 

<Request 

<Request 


GNews ▼ ^ BBC News ^ G Gmail Drive Calendar Courses PEP 8 


'http://localhost:5000/search4' [POST]> {'i', 'e'} 

'http://localhost:5000/search4' [POST]> {'i', 'e', 'u', 'a'} 

'http://localhost:5000/search4' [P0ST]> {'a'} 



Pa-ta abou-t v-e^uest 
o\)\tci has been savcd m w 
|o^, but -for sowc v-cason ibc 
web bvowsev is v-c-f^sm^ “to 

v-e*dev i*b oy\ sdvccy\- 


{y PA V Flask 


» 



The explanation as to why the request data is not rendering is subtle, and the fact 
that Firefox has highlighted the request data in red helps in understanding what 5 s 
going on. It appears there’s nothing wrong with the actual request data. However, 
it seems that the data enclosed in angle brackets (< and >) is upsetting the browser. 
When browsers see an opening angle bracket, they treat everything between that 
bracket and the matching closing angle bracket as an HTML tag. As <Request> 
is not a valid HTML tag, modern browsers simply ignore it and refuse to render 
any of the text between the brackets, which is what’s happening here. This solves 
the mystery of the disappearing request data. But we stili want to be able to see 
this data when we view the log using /viewlog. 

What we need to do is somehow teli the browser not to treat the angle brackets 
surrounding the request object as an HTML tag, but treat them as plain-text 
instead. As luck would have it, Flask comes with a function that can help. 
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storing & manipulating data 


Ifs Time to Escape (Your Pata) 


When HTML was first created, its designers knew that some web page designers 
would want to display angle brackets (and the other characters that have special 
meaning to HTML). Gonsequently, they came up with the concept known as 
escaping : encoding HTML’s special characters so that they could appear on a 
webpage but not be interpreted as HTML. A series of translations were defined, 
one for each special character. It 5 s a simple idea: a special character such as < is 
defined as & 11;, while > is defined as &gt;. If you send these translations instead 
of the raw data, your web browser does the right thing: it displays < and > as 
opposed to ignoring them, and displays all the text between them. 

Flask includes a function called escape (which is actually inherited fromJinja2). 
When provided with some raw data, escape translates the data into its HTML- 
escaped equivalent. Let’s experiment with escape at the Python >>> prompt to 
get a feel for how it works. 

Begin by importing the escape function from the flask module, then call 
escape with a string containing none of the special characters: 



Flask's Markup object is 
text that has been marked 
as being safe within an 
HTML/XML context. Markup 
inherits from Pythorfs 
built-in Unicode string, 
and can be used anywhere 
you'd use a string. 


|r*port “the 

-Cundtion- 


»> from flask import escape 
»> escape ( f This is a Request') 
Markup('This is a Request') 


The escape function returns is a Markup object , which—for all intents and 
purposes—behaves just like a string. When you pass escape a string containing 
any of HTML’s special characters, the translation is done for you, as shown: 


L 


As in the previous example (above), you can also treat this markup object as if it’s 
a regular string. 

If we can somehow arrange to call escape on the data in the log file, we should 
be able to solve the problem we currently have with the nondisplay of the request 
data. This should not be hard, as the log file is read “in one go” by the view_ 
the_log function before being returned as a string: 




Ko change 


»> escape ( f This is a <Request> ! ) 
Markup( 1 This is a &lt;Request&gt; 1 ) 



Use Wape" 
with a norn*al 
string. 


Use “escape” with 

d string Containing 

some special 

eharacters. 


The special 
characters have 
been esdaped (i.e-, 
translated). 


Here s our log 
data (as a 
string)- 


@app.route('/viewlog 1 ) 
def view_the_log() -> str: 
with open('vsearch.log') 

- -p contents = log. read () 

return contents 


as log: 


To solve our problem, all we need to do is call escape on contents. 
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escape raw data 


Viewiwg the Entire Log iw Your Webapp 

The change to your code is trivial, but makes a big difference. Add escape to the 
import list for the f lask module (at the top of your program), then call escape 
on the string returned from calling the j oin method: 


from flask import Flask, render template, request, escape 


@app.route('/viewlog') 
def view_the_log() -> str: 
with open('vsearch.log') 
contents = log.read() 
return escape(contents) 


as log: 




CairWape 

ov\ the 

veturhed 

s-brihg. 


Add to 

the 

hst- 


Amend your program to import and call escape as shown above, then save your code (so that your 
webapp reloads). Next, reload the /viewlog URL in your browser. AII of your log data should now 
appear on screen. Be sure to view the HTML source to confirm that the escaping is working. Here’s 
what we saw when we tested this version of our webapp with Chrome: 



0 127.0.0.1:5000/viewlog x 

D 127.0.0.1:5000/viewlog <£? 


<Request 'http://localhost:5000/search4' |POST]> {'i', 'e'} <Request 'http://localhost:5000/search4' [POST]> {'i', 'e', 'u', 
'a'} <Request 'http://localhost:5000/search4' [POST]> {'a'} 


AII the data 
tvoim the lo$ 
-file *»s now 
apfcavm^.. 


Q 127.0.0.1:5000/viewlog x Q view-source:127.0.0.1 :50 x 



C D view-source:127.0.0.1:5000/viewlog 


☆ 


&lt;Request &#39;http://localhost:5000/search4&#39; [POST]&gt; {&#39;i&#39;, &#39;e&#39;} 
&lt;Request &#39;http://localhost:5000/search4&#39; [POST]&gt; {&#39;i&#39;, &#39;e&#39;, 
&#39;u&#39;, &#39;a&#39;} 

&lt;Request &#39;http://localhost:5000/search4&#39; [POST]&gt; {&#39;a&#39;} 



A?±L d ihe e f ca?i ^ is 
nlthough—to be honest—the 

re^uest data doesh't veally teli U s 
r*u£h, does it? 
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storing & manipulating data 


learwmg More About the Request Object 

The data in the log file relating to the web request isn’t really all that useful. 
Here’s an example of what’s currently logged; although each logged resuit is 
different, each logged web request is showing up as exactly the same: 


fcdth lo 99 ed v»eb — 

ve<\uest is the sa»»e- p 

<Request 'http://localhost:5000/search4' [POST]> 

<Request 'http://localhost:5000/search4' [POST]> 
<Request 'http://localhost:5000/search4’ [POST]> 


We’re logging the web request at the object level, but really need to be looking 
inside the request and logging some of the data it contains. As you saw earlier in 
this book, when you need to learn what something in Python contains, you feed it 
to the dir built-in to see a list of its methods and attributes. 


{'i', 'e'} 

{'i', 'e', 'u', 'a'} 

{ 'a' } 


loaqcd 

resuit is 

di-MWeht. 



Let 5 s make a small adjustment to the log_request function to log the output 
from calling dir on each request object. It’s not a huge change...rather than 
passing the raw req as the first argument to print, let’s pass in a stringified 
version of the resuit of calling dir (req) . Here’s the new version of loq 
request with the change highlighted: 



def log_request (req: ' flask_requestres : str) -> None: 
with open(’vsearch.log ', 'a') as log: 

print(str(dir(req)), res, file=log) 



,1 y ^ ec l * whidh F^dudes a list, a*d the* we 
stvmgi+y the list by P assi* 9 the list to "str". The resulti* 9 

strmg .s the* saved to the log tile alo* 9 with the value of Ves". 


Lefs try out this new logging code to see what difference it makes. Perform the following steps: 

1. Amend your copy of log_request to match ours. 

2. Save vsearch4log .py in order to restart your webapp. 

3. Find and delete your current vsearch. log file. 

4. Use your browser to enter three new searches. 

5. View the newly created log using the /viewlog URL. 

Now: have a good look at what appears in your browser. Does what you now see help at all? 
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str dir req 



Here’s what we saw after we worked through the five steps from the bottom of the last page. We’re 
using Safari (although every other browser shows the same thing): 


ED 


V 127.0.0.1:5000/viewlog 


C 




3 


TWis all looks 
kmda ^cssy. 

But 1 0<M 

t\ost\y kcvc s 
-the vcsul*U 

of OY\t O-f 

{\\c scavdkcs 
we pev£ovr*ed- 


+ 


['_class_'_delattr_'_dict_'_dir_'_doc_'_enter_'_eq_'_exit_'_format_'_ge_ 

'_getattribute_'_gt_'_hash_'_init_'_le_'_lt_'_module_'_ne_'_new_','_reduce_ 

'_reduce_ex_'_repr_'_setattr_'_sizeof_'_str_'_subclasshook_'_weakref_'_get_file_stream', 

'_get_stream_for_parsing', '_is_old_module', '_load_form_data', '_parse_content_type', '_parsed_content_type', 
'accept_charsets', 'accept_encodings', 'accept_languages', 'accept_mimetypes', 'access_route', 'application', 'args', 
'authorization', 'base_url', 'blueprint', 'cache_contror, 'charset', 'close', 'content_encoding', 'content_length', 'content_md5', 
'content_type', 'cookies', 'data', 'date', 'dict_storage_class', 'disable_data_descriptor', 'encoding_errors', 'endpoint', 'environ', 
'files', 'form', 'form_data_parser_class', 'from_values', 'full_path', 'get_data', 'get_json', 'headers', 'host', 'host_url', 'if_match', 
'if_modified_since', 'if_none_match', 'if_range', 'if_unmodified_since', 'input_stream', 'is_multiprocess', 'is_multithread', 
'is_run_once', 'is_secure', 'is_xhr', 'json', 'list_storage_class', 'make_form_data_parser', 'max_content_length', 
'max_form_memory_size', 'max_forwards', 'method', 'mimetype', 'mimetype_params', 'module', 'on_json_loading_failed', 
'parameter_storage_class', 'path', 'pragma', 'query_string', 'range', 'referrer', 'remote_addr', 'remote_user', 'routing_exception', 
'scheme', 'script_root',' s b-* 11 -"i.'stmam' Jt"led hosts', 'uri', 'url_charset', 'url_root', 'url_rule', 'user_agent', 'values', 

'view_args', 'wantiform_data_pa^^j / {'x', 'y'} ['_ class_','_delattr_','_dict_','_dir_','_doc_','_enter_', 

t_','_format_','_gc_'f_getattribute_','_gt_','_hash_','_init_','_le_','_lt_', 

j_','_ne_','_new_','_reduce_','_reduce_ex_','_repr_','_setattr_','_sizeof_','_str_', 

_subclasshook_','_weakref_', '_get_file_stream', '_get_stream_for_parsing', '_is_old_module', '_load_form_data', 

'_parse_content_type', '_parsed_content_type', 'accept_charsets', 'accept_encodings', 'accept_languages', 

'accept_mimetypes', 'access_route', 'application', 'args', 'authorization', 'base_url', 'blueprint', 'cache_control', 'charset', 

'close', 'content_encoding', 'content_length', 'content_md5', 'content_type', 'cookies', 'data', 'date', 'dict_storage_class', 
'disable_data_descriptor', 'encoding_errors', 'endpoint', 'environ', 'files', 'form', 'form_data_parser_class', 'from_values', 
'full_path', 'get_data', 'get_json', 'headers', 'host', 'host_url', 'if_match', 'if_modified_since', 'if_none_match', 'if_range', 
'if_unmodified_since', 'input_stream', 'is_multiprocess', 'is_multithread', 'is_run_once', 'is_secure', 'is_xhr', 'json', 
'list_storage_class', 'make_form_data_parser', 'max_content_length', 'max_form_memory_size', 'max_forwards', 'method', 
'mimetype', 'mimetype_params', 'module', 'on_json_loading_failed', 'parameter_storage_class', 'path', 'pragma', 
'query_string', 'range', 'referrer', 'remote_addr', 'remote_user', 'routing_exception', 'scheme', 'script_root', 'shallow', 'stream', 
'trusted hosts','uri','uri charset','uri root','uri rule','user agent' ,'values', 'view args','want form data parsedj {'u', 'i', 'e', 



What'$ all thi$, thew? 

You can just about pick out the logged results in the above output. The rest of the 
output is the resuit of calling dir on the request object. As you can see, each request 
has a lot of methods and attributes associated with it (even when you ignore the dunders 
and wonders). It makes no sense to log all of these attributes. 

We took a look at all of these attributes, and decided that there are three that we think 
are important enough to log: 

req. form: The data posted from the webapp’s HTML form. 

req. remote_addr: The IP address the web browser is running on. 

req. user_aqent: The identity of the browser posting the data. 

Let’s adjust loq_request to log these three specific pieces of data, in addition to the 
results of the call to search41etters. 
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storing & manipulating data 


Loggiwg Specif ic Web Request Attributes 

As you now have four data items to log—the form details, the remote IP address, the 
browser identity, and the results of the call to search41etters—a first attempt at 
amending log_request might resuit in code that looks like this, where each data 
item is logged with its own print call: 


def log request(req:'flask requestA resistr) -> None: 


Lo$ eac.h data 
with its 


u 


r 


own fv-m 
statemcnt 


with open('vsearch.log', 'a') as log 

print(req.form, file=log) 
print(req.remote_addr, file=log) 
print(req.user_agent, file=log) 
print(res, file=log) 


This code works, but it has a problem in that each print call appends a newline 
character by default, which means there are four lines being logged per web request. 
Here’s what the data would look like if the log file used the above code: 

Thev-es a line ot data bv- 
eatK vemote IP addvess. 



The data as entered in-fco 
the HTML tov-m appears 

( its own line. BTlV: the 
"(mmutable/VtultiDiet" is a 
Flask—spedi-pid veirsioh ot 
Python s didtionairy (and it 
wonks in the sa*e way). 


ImmutableMultiDict([('letters', 'aeiou'), ('phrase', 'hitch-hiker')]) 

127.0.0.1 

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) ... Safari/601.4.4 

{'i', 'e'} 

ImmutableMultiDict([('letters 1 , 'aeiou ')a ('phrase', 'life, the universe, and everything')]) 
127.0.0.1 



Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) .. 

{'a', 'e', 'i', 'u'} 

ImmutableMultiDict([('letters', 'xyz'), ('phrase', 

127.0.0.1 

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) .. 

{'x', 'y'} 

— The resul*ts o-P "the £all seard.h^He^tars^ are 
dlearly shown Uac\\ on its own line). 


Safari/601.4.4 


galaxy')]) 


Safari/601.4.4 



The browser 
is identi-fied 
or\ its own 
line. 


There’s nothing inherently wrong with this as a strategy (as the logged data is easy for 
us humans to read). However, consider what you’d have to do when reading this data 
into a program: each logged web request would require four reads from the log file— 
one for each line of logged data. This is in spite of the fact that the four lines of data 
refer to one single web request. As a strategy, this approach seems wasteful. It would be 
much better if the code only logged one line per web request. 
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dealing with newlines 


log a Single Line of Pelimited Pata 


A better logging strategy may be to write the four pieces of data as one line, while using 
an appropriately selected delimiter to separate one data item from the next. 

Choosing a delimiter can be tricky, as you don’t want to choose a character that might 
actually occur in the data you’re logging. Using the space character as a delimiter is next 
to useless (as the logged data contains lots of spaces), and even using colon (:), comma 
(,), and semicolon (;) may be problematic given the data being logged. We checked 
with the programmers over at Head First Labs , and they suggested using a vertical bar (|) 
as a delimiter: it’s easy for us humans to spot, and it 5 s unlikely to be part of the data we 
log. Let 5 s go with this suggestion and see how we get on. 

As you saw earlier, we can adjust print 5 s default behavior by providing additional 
arguments. In addition to the file argument, there 5 s the end argument, which allows 
you to specify an alternate end-of-line value over the default newline. 

Let 5 s amend log_request to use a vertical bar as the end-of-line value, as opposed to 
the default newline: 



Think of a delimiter as a 
sequence of one or more 
characters performing 
the role of a boundary 
within a line of text.The 
classic example is the 
comma character (,) as 
used in CSV files. 


def log_request (req: ' flask_request' , res: str) -> None: 
with open('vsearch.log ', 'a') as log: 

print(req.form, file=log, end='|') 
print(req.remote_addr, file=log, end='|') 
print(req.user_agent, file=log, end='|') 
print(res, file=log) 


This works as expected: each web request now results in a single line of logged data, 
with a vertical bar delimiting each logged data item. Here’s what the data looks like in 
our log file when we used this amended version of log_request: 


Eadh o-f 

-fchesc M pv-iht w 
sta-fcerKCh^fcs 

reptans the 
dc- fauffc hCwlihC 
wi-th a ver-tieal 
bav. 



Eadb web ve«\uest is wrrbten to i*U ovm l'mc (v/hidh we ve 
v/ov-d-wv-afped m ov-dev- “to -fit oy\ “tbis fd^e)- 

ImmutableMultiDict([('letters', 'aeiou'), ( 'phrase ', 'hitch-hiker')]) |127.0.0.1|Mozilla/5.0 

(Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 

Safari/601.3.9|{'e', 'i'} 

ImmutableMultiDict([('letters', 'aeiou ')a ('phrase', 'life, the universe, and everything')])|12 
7.0.0.1|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) 
Version/9.0.2 Safari/601.3.9|{'e', 'u', 'a', 'i'} 

ImmutableMultiDict([('letters', 'xyz'), ('phrase', 'galaxy')])|127.0.0.1|Mozilla/5.0 

(Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 

Safari/601.3. 'x'} 

“ Theve 'ncrt -Bvree 

vjeb ve*\<Aests) so M 

n,ea ? s we have lo 99 ed ot daia t-le 


Did you spo-t -the veir-tidal bavs used as 
delimi-teirs? There are ihree bav-s, whidh 




|'mc. 
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Ohe Final Change to Our Logging Code 

Working with overly verbose code is a pet peeve of many Python programmers. Our 
most recent version of log_request works fine, but it’s more verbose than it needs 
to be. Specifically, it feels like overkill to give each item of logged data its own print 
statement. 

The print function has another optional argument, sep, which allows you to 
specify a separation value to be used when printing multiple values in a single call to 
print. By default, sep is set to a single space character, but you can use any value 
you wish. In the code that follows, the four calls to print (from the last page) have 
been replaced with a single print call, which takes advantage of the sep argument, 
setting it to the vertical bar character. In doing so, we negate the need to specify a 
value for end as the print 5 s default end-of-line value, which is why all mentions of 
end have been removed from this code: 

def log_request (req: ’ flask_request' , res: str) -> None: 
with open('vsearch.log', 'a’) as log: 

print(req.form, req.remote addr, req.user agent, res, file=log, sep=’ 


Onh/ one "prih-t 

call msfcead ot 
•Powr 




Yes, this line breaks a PEP 8 guideline. 

Some Python programmers frown at this last line 
of code, as the PEP 8 Standard specifically warns 
against lines longer than 79 characters. At 80 
characters, our line of code is pushing this guideline 
a little , but we think it’s a reasonable trade-off given 
what we’re doing here. 

Remember: striet adherence to PEP 8 is not an 
absolute must, as PEP 8 is a style guide : not an 
unbreakable set of rules. We think we’re good to go. 


e- 


i 
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log more data 



Let’s see what difference this new code makes. Adjust your log_request function to look 
like this: 


def log_request(req: ’flask_request', res: str) -> None: 
with open('vsearch.log', 'a’) as log: 

print(req.form, req.remote addr, req.user agent, res, file=log, sep='|') 


Then perform these four steps: 

1. Save vsearch4log. py (which restarts your webapp). 

2. Find and delete your current vsearch. log file. 

3. Use your browser to enter three new searches. 

4. View the newly created log using the /viewlog URL. 

Have another good look at your browser display. Is this better than before? 



Having completed the four steps detailed in the above exercise, we ran our latest tests using Chrome. 
Here’s what we saw on screen: 


• 0 127.0.0.1:5000/Viewlog X 

• 

=f> C 

D 127.0.0.1:5000/viewlog 



ImmutableMultiDict([('phrase', 'hitch-hiker'), ('letters', 'aeiou')])ll27.0.0.1IMozilla/5.0 (Macintosh; Intel Mac OS X 
10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36l{'e', 'i'} 
ImmutableMultiDict([('phrase', 'life, the universe, and everything'), ('letters', 'aeiou')])ll27.0.0.1IMozilla/5.0 (Macintosh; 
Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36l{'e', 'a', 'u', 
'i'} ImmutableMultiDict([('phrase', 'galaxy'), ('letters', 'xyz')])ll27.0.0.1IMozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) 
Apple WebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36l{'x', 'y'} 

t 

VtWs tevbmly a lot less data out ? «.t 

vrodutd by tbc prcvious vevsio* ok lo^c«\ucst , * 
is stili a bit of a mess...a»d it's hard to <*t the W 
f iedes ot lo^ed data (eve* with all those vertical ha« as 

dclimitcv-s). 
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Frow Raw Data to Readable Output 

The data displayed in the browser window is in its raw form. Remember, we perform 
HTML escaping on the data as read in from the log file but do nothing else before 
sending the string to the waiting web browser. Modern web browsers will receive the 
string, remove any unwanted whitespace characters (such as extra spaces, newlines, 
and so on), then dump the data to the window. This is what 5 s happening during our 
Test Drive. The logged data—all of it—is visible, but it 5 s anything but easy to read. We 
could consider performing further text manipulations on the raw data (in order to 
make the output easier to read), but a better approach to producing readable output 
might be to manipulate the raw data in such a way as to turn it into a table: 


ImmutableMultiDict([('phrase', 'hitch-hiker'), ('letters', 'aeiou')])|127.0.0.1|Mozilla/5.0 

(Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 
Safari/537.36|{'e', 'i'} ImmutableMultiDict([('phrase', 'life, the universe, and 

everything'), ('letters', 'aeiou')])|127.0.0.1|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) 
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36| {'e ' , 'a', 'u', 

'i'} ImmutableMultiDict([('phrase', 'galaxy' ) , ('letters', 'xyz')]) |127.0.0.1|Mozilla/5.0 

(Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 
Safari/537.36|{'x', 'y'} 

Ca* *,e take tW.s Ureadable) ra* data- 


Form Data 

Remote_addr 

User_agent Results 

Immu-tablcAlul-tiPid-tfCrphv-asc^ 
Vrtdb—bikevO, Oettevs\ WiouO]) 

111.0.0.1 

Mozilla/5.0 (Matintosb; {'e', Y} 

Intel /Hat 0£ X IO II 1) 
A^leWebkit/^lli 04H7HU 
like ^etko) CKvome/ 17 0.Z5Z/> 

.I0(> Satav- ■,/Kl.tt 

tbc univev-se, and evevytbmg ), 
aeiou)]) 

1 11 .o.o.l 

Mozilla/5.0 (Matintosb; {'e', 'a', V, Y} 

Intel /Hat 0£ X >0 II 1) 
AffleWeb^it/^73t 04H7HU 
like £jetko) Chv-ome/^r70-2.^2.t 
■I0t> Satavi/ 1 ?} l.V> 

| mmu-tab 1 eM ul ti Pidtf trpbvase^ 
jala*y), OettevV, Vyz/)]) 

111 .o.o.l 

/Hozilla/5.0 (/Hatintosb; {V, 'yl 

Intel /Hat OS x IO II 2.) 
/\ffleWeb^it/^73b (kHT/HU 
like ^etko) CKvome/ 17 0.Z5Z/> 

•lOt Satavi/ 1 ?}!.}*. 


J 


a table that looks like tbisi 


> 


If our webapp could perform this transformation, then anyone could view the log data 
in their web browser and likely make sense of it. 
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deja vu? 


Poes This Remind You of Anytbing? 

Take another look at what you are trying to produce. To save on space, we’re 
only showing the top portion of the table shown on the previous page. Does 
what you’re trying to produce here remind you of anything from earlier in 
this book? 



Yes. That does look like something we’ve seen before. 

At the end of Ghapter 3, recall that we took the table of data below 
and transformed it into a complex data structure—a dictionary of 
dictionaries: 


Name 

Gender 

Occupatiori 

Home Planet 

Ford Pre-fedt 

Male 

Reseavdhev 

Bctclgcusc Seve* 

/Vthur Dent 

Male 

£a*dwidh-Makev 

EavtK 

Tvidia MdMilla» 

Ferale 

Mathc^atidia^ 

EavtK 

Mawin 

Unknoym 

Paranoid A^dv-oid 

Unknovm 


The shape of this table is similar to what we’re hoping to produce above, 
but is a dictionary of dictionaries the right data structure to use here? 
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U$e a Piet of Pict$...or Sometbing Else? 

The table of data from Chapter 3 fit the dictionary of dictionaries model because 
it allowed you to quickly dip into the data structure and extract specific data. For 
instance, if you wanted to know Ford Prefecfs home planet, all you had to do was 
this: 


ss Ford^ data- 


people [ 1 Ford' ] [' Home Planet 



] 


When it comes to randomly accessing a data structure, nothing beats a dictionary 
of dictionaries. However, is this what we want for our logged data? 

Lef s consider what we currently have. 


-theb ex-tra£-fc the 
value assodia-ted wi-th 
■fche Horne Planci" key. 


Take a closer look at the logged data 


Remember, every logged line contains four pieces of data, each separated by 
vertical bars: the HTML form 5 s data, the remote IP address, the identity of the 
web browser, and the results of the call to search41etters. 


Here’s a sample line of data from our vsearch.log file with each of the 
vertical bars highlighted: 


The -Covm data 



"The |P address o-P 
the remote madhi^e 


ImmutableMultiDict([('phrase ', 'galaxy'), ( 'letters ', 'xyz')]) |127.0.0.1|Mozilla/5.0 (Macintosh; Intel 

Mac OS X 10 11 2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36| {'x 1 y f } 


The web browser^s 
identity strihej 



The results 
oJf the tali to 

“searth^letters’ 



When the logged data is read from the vsearch.log file, it arrives in your code 
as a list of strings thanks to our use of the readlines method. Because you 
probably won’t need to randomly access individual data items from the logged 
data, converting the data to a dictionary of dictionaries seems like a bad move. 
However, you need to process each line in order , as well as process each individual 
data item within each line in order. You already have a list of strings, so you’re half- 
way there, as it’s easy to process a list with a for loop. However, the line of data 
is currently one string, and this is the issue. It would be easier to process each line 
if it were a list of data items, as opposed to one large string. The question is: is it 
possible to convert a string to a list? 
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split that join 


WhaTs Jomed Together Can Pe Split Apart 


You already know that you can take a list of strings and convert them to a single string 
using the “join trick.” Let’s show this once more at the >>> prompt: 


A list o-f 
indiviclual s-b-inas 


»> names = ['Terry', 'John', 'Michael', 'Graham', 'Eric'] 

»> pythons = ' | join (names) _The Vm-tvitk" m atW. 

»> pythons * . , . 

' Terry | John | Michael | Graham | Eric 1 ---« , ■*,' Wlu * sfo-mg -(Vom -the 

rr 

a«d delimited by a vevbeal bair 


Thanks to the “join trick,” what was a list of strings is now a single string, with each 
list item separated from the next by a vertical bar (in this case). You can reverse this 
process using the split method, which comes built in to every Python string: 


»> individuals = pythons . split ( 1 | f ) 

»> individuals 

[ 1 Terry 1 , 1 John 1 , 1 Michael 1 , 1 Graham 1 , 1 Eric 1 ] 


Takc -the stving and split 
it into a list using the 
fjiveh delimitev. 


And now we 
ave badk to ouv 
list sbrinjs. 


(rettiwg to a list of lists from a list of strings 

Now that you have the split method in your coding arsenal, let’s return to the data 
stored in the log file and consider what needs to happen to it. At the moment, each 
individual line in the vsearch.log file is a string: 


ImmutableMultiDict([ ('phrase ', 'galaxy'), ( 'letters ', 'xyz')]) |127.0.0.1|Mozilla/5.0 (Macintosh; Intel 

Mac OS X 10 11 2) AppleWebKit/537.36 (KHTML^ like Gecko) Chrome/47.0.2526.106 Safari/537.36|{'x ', 'y'} 


^~he vaw data 

j 


Your code currently reads all the lines from vsearch.log into a list of strings called 
contents. Shown here are the last three lines of code from the view_the_log 
function, which read the data from the file and produce the large string: 

• • • 

with open('vsearch.log') as log: 

contents = log. readlines () <<■ 
return escape(’'.join(contents)) 


Open the log -Pi|< 



...and vead all the 


The last line of the view_the_log function takes the list of strings in contents 
and concatenates them into one large string (thanks to join). This single string is then 
returned to the waiting web browser. 


lines ot log data 
mto a list talled 

“dontents*- 


If contents were a list of lists instead of a list of strings , it would open up the 
possibility of processing contents in order using a for loop. It should then be 
possible to produce more readable output than what we’re currently seeing on screen. 
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storing & manipulating data 


When Should the Cowversiow Occur? 

At the moment, the view_the_log function reads all the data from the log file 
into a list of strings (called contents). But we’d rather have the data as a list of 
lists. The thing is, when’s the “best time” to do this conversion? Should we read 
in all the data into a list of strings, then convert it to a list of lists “as we go,” or 
should we build the list of lists while reading in each line of data? 



The fact that the data is already in contents (thanks to our use of the 
readlines method) shoukkft blind us to the fact that weVe already looped 
through the data once at this point. Invoking readlines may only be a single 
call for us, but the interpreter (while executing readlines) is looping through 
the data in the file. If we then loop through the data again (to convert the strings 
to lists), we’re doubling the amount of looping that’s occurring. This isn’t a big 
deal when there 5 s only a handful of log entries...but it might be an issue when the 
log grows in size. The bottom line is this: if we can make do by only looping once, then 
lefs do so! 
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read split escape 


Processing Pata: What We Already Know 


Earlier in this chapter, you saw three lines of Python code that processed the lines 
of data in the todos . txt file: 


Ope* 


the file* 




Assijh 

“the -Pii t 

s tream to 

a vaHable. 


with open('todos.txt') as tasks 
for chore in tasks 
print (chore, end= 


nd='') 


Per-Porm some processi 


or\t lihe at a t 




ime. 


You Ve also seen the split method, which takes a string and converts it to a list 
of strings based on some delimiter (defaulting to a space, if none is provided). In 
our data, the delimiter is a vertical bar. Let 5 s assume that a line of logged data is 
stored in a variable called line. You can turn the single string in line into a list 
of four individual strings—using the vertical bar as the delimiter—with this line 
of code: 


This is the *ame 
Jc the inewly 
ereated list 


WeVe usi*g a vertical bar 
tbe delimiter 


four strings = line.split('| 


Use “split” to break the string 'mto a 

list ot substrir^s. 


As you can never be sure whether the data youVe reading from the log file is 
free of any characters that have special meaning to HTML, you Ve also learned 
about the escape function. This function is provided by Flask, and converts any 
string 5 s HTML special characters into their equivalent escaped values: 

»> escape ( f This is a <Request> ! ) 

Markup( 1 This is a &lt;Request&gt; 1 ) 


Use "esta^e” wth 
a fcontainin^ 

HTML spef.al 

thavattevs- 


And, starting way back in Chapter 2, you learned that you can create a new list by 
assigning an empty list to it ([ ]). You also know that you can assign values to the 
end of an existing list by calling the append method, and that you can access the 
last item in any list using the [ -1 ] notation: 

_Create a hew, empty list 

called “names". 

»> names = [] 

»> names . append ( 1 Michael 1 ) } _/\dd some data to the end 

»> names . append (' John') J ^ ot the existi*) list 

»> names [-1] 

' John ' /^dtess the last item in the names list- 


Armed with this knowledge, see if you can complete the exercise on the next page. 
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Here is the view the log functiorVs current code: 


@app. route (' /viewlog') 
def view_the_log() -> str: 

with open('vsearch.log') as log: 

contents = log.readlines() 
return escape(’’.join(contents)) 


The fcvst 

two lines 

ven^ain 

undhan^cd- 


This code reads the data from the log file into a list of strings. Your job is to 
convert this code to read the data into a list of lists. 

Make sure that the data written to the list of lists is properly escaped, as 
you do not want any HTML special characters sneaking through. 

Also, ensure that your new code stili returns a string to the waiting web 
browser. 

WeVe got you started—fili in the missing code: 


@app.route('/viewlog') 

def view the log() -> 'str': 



J\dd youv new 
dode hev-e- 


The tundtion 

stili v-etuvns -^ return str (contents) 

a stvm<y 


Take youv time hev-e. Feel tvee to e%pev-iment at the »> shell as needed, 
a*d don t wov-v-y it you $et studk—it ; s to tlip the pa$e and look at the 


soluti 


ion- 
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list oflists 


wr /T.r: 

J k Solution 


Here is the view the log functiorVs code: 


@app.route('/viewlog') 
def view_the_log() -> str: 

with open('vsearch.log’) as log: 

contents = log.readlines() 
return escape(''.join(contents)) 

Your job was to convert this code to read the data into a list of lists. 

You were to ensure that the data written to the list of lists is properly 
escaped, as you do not want any HTML special characters sneaking 
through. 

You were also to ensure that your new code stili returns a string to the 
waiting web browser. 

We'd started for you, and you were to fili in the missing code: 


@app.route('/viewlog') 

Cv-ca-tc a new, def view_the_log() -> 'str': 

C»nr\ p*ty list dallcd ^ ^ ^ 

Wwb". ^-^ iokois - □. fj? 'f 9 ^ * si 9» * 

. .— 3 st^rca^ ca\\td log 

y/ith openCvseav-dlvIogO as log: ^ 

Loof “Ihvough ... 

eaeh 1'rne m -the- ...%Jfl:t.1?$ a j , 

"I05" -file stv-ear». MH» 'Wttt? ^ 

r . -__-^ fov item 'm l'mesp|i-t('|0 : ■/--7 OIA 

Split the line (based or> . r'7'r 'n ..... remembev to 

the vevtieal baO, the» tali "estape"? 

ptrodess eaeh iter» i» the K d a dd , . ,, 

vesultihjj "split list”. return str (contents) i i. e e ^taped data 

1 to the e»d 0+ the list at 

the e»d of u dohte»ts”. 


Don’t worry if this line of code from the above rewrite of the view_the 
function has your head spinning: 


log 


contents[-1].append(escape(item)) 



The trick to understanding this (initially daunting) line is to read it from the inside 
out, and from right to left. You start with the item from the enclosing for loop, 
which gets passed to escape. The resulting string is then appended to the list at 
the end ([ -1 ]) of contents. Remember: contents is itself a list of lists. 


Read this eode &or» 
f^e inside out, a»d 
W» nght to left 
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Go ahead and change your view_the_log function to look like this: 


@app. route (' /viewlog') 
def view_the_log() -> 'str': 
contents = [] 

with open('vsearch.log') as log: 
for line in log: 

contents.append([]) 

for item in line.split('|'): 

contents[-1].append(escape(item)) 
return str(contents) 


Save your code (which causes your webapp to reload), then reload the /viewlog URL in your browser. 
Here’s what we saw in ours: 


ED 


S 127.0.0.1:5000/viewlog 


c 


fi 


3 


+ 


[[Markup('ImmutableMultiDict([('letters', 'aeiou'), ('phrase', 'hitch-hiker')])'), MarkupC 127.0.0.1'), Markup('Mozilla/5.0 
(Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9'), Markup('{'e', 
'i'}\n')L [Markup('ImmutableMultiDict([('letters', 'aeiou'), ('phrase', 'life, the universe, and everything')])'), MarkupC 127.0.0.1'), 
Markup('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 
Safari/601.3.9'), Markup('{'e', 'u', 'a', 'i'}\n')], [Markup('ImmutableMultiDict([('letters', 'xyz'), ('phrase', 'galaxy')])'), 

MarkupC 127.0.0.1'), Markup('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) 
Version/9.0.2 Safari/601.3.9'), Markup('{'y', 'x’}\n')]] 



The da*ta ,s bafck 

oki -tVie stveen • <w ,s 


Take a closer look at the output 

At first glance, the output produced by this new version of view_the_log 
looks very similar to what you had before. But it isn’t: this new output is a list 
of lists, not a list of strings. This a crucial change. If you can now arrange 
to process contents using an appropriately designedjinja2 template, you 
should be able to get pretty close to the readable output required here. 
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<table> with jinja2 


Generate Readable Output With HTML 

Recall that our goal is to produce output that looks better on screen than the 
raw data from the last page. To that end, HTML comes with a set of tags 
for defining the content of tables, including: <table>, <th>, <tr>, and 
<td>. With this in mind, let’s take another look at the top portion of the 
table we’re hoping to produce once more. It has one row of data for each line 
in the log, arranged as four columns (each with a descriptive title). 

You could put the entire table within an HTML <table> tag, with each row 
of data having its own <tr> tag. The descriptive tities each get <th> tags, 
while each piece of raw data gets its own <td> tag: 

£adh “the 
headwgs goes 
i*thm 3 <“th> “tajy 




Here's a quick review of the HTML 
table tags: 

<table>: A table. 

<tr>: A row of table data. 

<th>: A table column heading. 
<td>: A table data item (cell). 

Each tag has a corresponding end 
tag: </table>, </tr>, 

</th>, and </td>. 


The ewbvt 

-table goes 
wVthm a 
<-table> 



Eaeh picee 
data goes with i k 
a <td> tag. 


£aeh v-ow m the 
table goes wiihw 
a <tv> tag- 


Whenever you find yourself needing to generate any HTML (especially 
a <table>), remember Jinja2. The Jinja2 template engine is primarily 
designed to generate HTML, and the engine contains some basic 
programming constructs (loosely based on Python syntax) that you can use to 
“automate” any required display logic you might need. 

In the last chapter, you saw how the Jinja2 { { and } } tags, as well as the 
{ % block % } tag, allow you to use variables and blocks of HTML as 
arguments to templates. It turns out the { % and % } tags are much more 
general, and can contain any Jinja2 statement , with one of the supported 
statements being a for loop construet. On the next page youTl find a new 
template that takes advantage of Jinja2 5 s for loop to build the readable 
output from the list of lists contained in contents. 
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Embed Pisplay Logic in Your Template 


Below is a new template, called viewlog . html, which can be used to transform 
the raw data from the log file into an HTML table. The template expects the 
contents list of lists to be one of its arguments. WeVe highlighted the bits 
of this template we want you to concentrate on. Note that Jinja2 5 s for loop 
construet is very similar to Python’s. There are two major differences: 

• There’s no need for a colon (:) at the end of the for line (as the % } tag 
acts as a delimiter). 


• The loop’s suite is terminated with {% endfor %}, as Jinja2 doesn’t 
support indentation (so some other mechanism is required). 

As you can see, the first for loop expects to find its data in a variable called 
the_row_titles, while the second for loop expects its data in something 
called the_data. A third for loop (embedded in the second) expects its data to 
be a list of items: 


You don’t have to create 
this template yourself. 
Download it from 

http://python.itcarlow. ie/ed2/. 


I Reapy Batte 


To ensuve ^ 
Consistent look 
and -feel> this 
-template inhevrb 
-fy-om the s3mC 

base template 

used throughout 

ouv webapp- 



The entire 
table $oes _ 
y/ithin 3 

<table> ta<y 



{% extends 'base.html' %} 

{% block body %} 

<h2>{{ the_title }}</h2> 

<table> 

<tr> 

{% for row_title in the_row_titles %} 
<th>{{row_title}}</th> 

{% endfor %} 

</tr> 

{% for log_row in the_data %} 

<tr> 

{% for item in log_row 
<td>{{item}}</td> 

{% endfor %} 




</tr> 
{% endfor 
</table> 

{% endblock %} 


} 



The desCv-iptWe 
tities (each 
y/ithin a <th> 

tag) get theiv- 
own vow (the 
<tv> tag)- 

Eaeh individual 
item o-P logged 
data is endosed 
with i n a <td> 
tag ; and each line 
-from the log tile 
has its oy/n <tr> 


Be sure to place this new template in your webapp’s templates folder prior to use. 
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work that template 


Producing Readable Output with Jiwja2 

As the viewlog . html template inherits from base . html, you need to 
remember to provide a value for the the_title argument and provide a list of 
column headings (the descriptive tities) in the_row_titles. And don’t forget 
to assign contents to the the_data argument. 

The view_the_log function currently looks like this: 


@app.route('/viewlog') 
def view_the_log() -> 'str': 
contents = [] 
with open('vsearch.log') 
for line in log: 


as log: 


We £uv\rehtly 
retui™ a 

s-tirihg to -the 
waitiKg web 
bvowsev". 


contents.append([]) 

for item in line.split( 1 |'): 

contents[-1].append(escape(item)) 
return str(contents) 


You need to call render_template on viewlog . html, and pass it values for 
each of the three arguments it expeets. Let’s create a tuple of descriptive tities and 
assign it to the_row_titles, then assign the value of contents to the_ 
data. We’ll also provide an appropriate value for the_title before rendering 
the template. 

With all of that in mind, let’s amend view_the_log (weVe highlighted the 
changes): 

Change the armotatio* to mdi 
@app. route (' /viewlog') J that HT/l/IL is beihg vctuv-hcd 

def view_the_log () -> 'html': v bustead o-P a stv-ing). 
contents = [] 

with open('vsearch.log') as log: 
for line in log: 

contents.append([]) 

for item in line.split('|'): 

contents[-1].append(escape(item)) 

Cv-eate a-tities = ('Form Data', 'Remote_addr', 'User_agent', 

tu\>le o£ return render_template (' viewlog. html' , 

desdriftWe the_title='View Log' , 

the_r ow_t i tles=tities, 
the data=contents,) 


Go ahead and make these changes to your view_the_log function and then 
save them so that Flask restarts your webapp. When you’re ready, view the log 
within your browser using the http://127.0.0.1:5000/viewlog URL. 


Remember* 
a tuple is a 
read-only list 


£a-fce 


'Results') 

Call t-chdcv-^-tcmpl 
providmg values 
ea<ib o-P -the templa 

airgumehts. 
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Here’s what we saw when we viewed the log using our updated webapp. The page has the same look 
and feel as all our other pages, so we are confident that our webapp is using the correct template. 

We’re pretty pleased with the resuit (and we hope you are too), as this looks very similar to what we 
were hoping to achieve: readable output. 


• • • < > e 


127.0.0.1:5000/viewlog 


6 


5L [+ 


View Log 


Form Data 

Remote_addr 

User_agent 

Results 

ImmutableMultiDict([('letters', 
'aeiou'), ('phrase', 'hitch- 
hiker')]) 

127.0.0.1 

Mozilla/5.0 
(Macintosh; Intel 

Mac OS X 10_11_2) 
AppleWebKit/601.3.9 
(KHTML, like Gecko) 
Version/9.0.2 
Safari/601.3.9 

{'e', T> 

ImmutableMultiDict([('letters', 
'aeiou'), ('phrase', 'life, the 
universe, and everything')]) 

127.0.0.1 

Mozilla/5.0 
(Macintosh; Intel 

Mac OS X 10_11_2) 
AppleWebKit/601.3.9 
(KHTML, like Gecko) 
Version/9.0.2 
Safari/601.3.9 

{'e', 'u', 
'a', 'i’> 

ImmutableMultiDict([('letters', 
'xyz'), ('phrase', 'galaxy')]) 

127.0.0.1 

Mozilla/5.0 
(Macintosh; Intel 

Mac OS X 10 11 2) 
AppleWebKit/601.3.9 
(KHTML, like Gecko) 
Version/9.0.2 
Safari/601.3.9 

{V, 

'x'> 



If you view the source of the above page—right-click on the page, then choose 
the appropriate option from the pop-up menu—youTl see that every single data 
item from the log is being given its own <td> tag, each line of data has its own 
<tr> tag, and the entire table is within a HTML <table>. 



Islot ov\Vj |S *this 

«wtfwt readable, tot 
it OjOoA, too- © 
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time for review 


The Current State of Our Webapp Code 

Let 5 s pause for a moment and review our webapp 5 s code. The addition of the logging 
code (log_request and view_the_log) has added to our webapp’s codebase, but 
everything stili fits on a single page. Here’s the code for vsearch4web.py displayed in 
an IDLE edit window (which lets you review the code in all its syntax-highlighted glory): 

# # # vsearch4web.py - /Users/paul/Desktop/_NewBook/ch06/webapp/vsearch4web.py (3.5.1) 


from flask import Flask, render_template, request, escape 
from vsearch import search41etters 

app = Flask (_name_) 

def log_request (req: 1 flask_request' , resistr) -> None: 
with open (' vsearch.log' , 'a') as log: 

print (req.form, req.remote_addr, req.user_agent, res, file=log, sep='|') 

@app.route( 1 /search4' , methods=[ 1 POST' ]) 
def do_search () -> ' html' : 

phrase = request.form[ 1 phrase 1 ] 

letters = request.form[ 'letters' ] 

title = 'Here are your resuits:' 

resuits = str (search41etters(phrase, letters)) 

log_request(request, results) 

return render_template( 'results.html' , 

the_title=title, 
the_phrase=phrase, 
the_letters=letters, 
the_results=results,) 

@app.route( '/ f ) 

@app.route( '/entry' ) 

def entry_page () -> 'html' : 

return render_template( ? entry.html' , 

the_title= 'Welcome to search41etters on the web!') 

@app.route( 1 /viewlog' ) 
def view_the__log () -> ' html' : 
contents = [] 

with open ( 'vsearch.log' ) as log: 
for line in log: 

contents.append([]) 

for item in line.split( '|' ): 

contents[-1].append(escape(item)) 
tities = ('Form Data', 'Remote_addr' , 'User_agent' , 'Results') 
return render_template ( ' viewlog. html' , 

the_title='View Log' , 
the_row_titles=titles, 
the_data=contents,) 

if _name = ' main_' : 

app.run(debug=T rue ) 


Ln: 2 Coi: 0 
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Askmg Questions of Your Pata 


Our webapp’s functionality is shaping up nicely, but are we any closer to 
answering the questions posed at the start of this chapter: How many requests 
have been responded to? Whafs the most common list of letters? Which IP addresses are 
the requests comingfrom? Which browser is being used the most? 

The last two questions can be somewhat answered by the output displayed 
by the /viewlog URL. You can teli where the requests are coming from (the 
Remote_addr column), as well as see which web browser is being used 
(the User_agent column). But, if you want to calculate which of the major 
browsers is used most by users of your site, that’s not so easy. Simply looking 
at the displayed log data isn’t enough; youdl have to perform additional 
calculations. 


The first two questions cannot be easily answered either. It should be ciear 
that further calculations must be performed here, too. 



Only write more code when you have to. 

If all we had available to us was Python, then, yes, 
we’d need to write a lot more code to answer these 
questions (and any others that might arise). After all, it 5 s 
fun to write Python code, and Python is also great at 
manipulating data. Writing more code to answer our 
questions seems like a no-brainer, doesn’t it? 

WelL.other technologies exist that make it easy to 
answer the sort of questions we’re posing without us 
having to write much more Python code. Specifically, 
if we could save the log data to a database, we could 
take advantage of the power of the database 5 s querying 
technology to answer almost any question that might 
arise. 

In the next chapter, youdl see what’s involved in 
amending your webapp to log its data to a database as 
opposed to a text file. 


All we have to do 
is write a bunch more 
code to perform these 
calculations, right? 


you are here ► 
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the code 


Chapter 6 s Code 
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In+eresting...according 
to this, we re much 
better off s+oring our 
data in a database. 


Storing data in a relational database system is handy. 

In this chapter, you’ll learn how to write code that interacts with the popular MySQL database 
technology, using a generic database API called DB-API. The DB-API (which comes 
Standard with every Python install) allows you to write code that is easily transferred from 
one database product to the next...assuming your database talks SQL. Although we’ll be 
using MySQL, there’s nothing stopping you from using your DB-API code with your favorite 
relational database, whatever it may be. Let’s see what’s involved in using a relational 
database with Python. There’s not a lot of new Python in this chapter, but using Python to talk 
to databases is a big deal, so it’s well worth learning. 


this is a new chapter 
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time for sql 


Patabase-Enabliwg Your Webapp 

The plan for this chapter is to get to the point where you can amend your webapp 
to store its log data in a database, as opposed to a text file, as was the case in the last 
chapter. The hope is that in doing so, you can then provide answers to the questions 
posed in the last chapter: How many requests have been responded to? WhaVs the most common 
list of letters? Which IP addresses are the requests comingfrom? Which browser is being used the 
most? 

To get there, however, we need to decide on a database system to use. There are lots 
of choices here, and it would be easy to take a dozen pages or so to present a bunch of 
alternative database technologies while exploring the pluses and minuses of each. But 
we’re not going to do that. Instead, we’re going to stick with a popular choice and use 
MySQL as our database technology. 

Having selected MySQL, here are the four tasks we’11 work through over the next 
dozen pages: 

Install the MySQL server 
Install a MySQL database driver for Python 
Create our webapps database and tables 
Create code to work with our webapps database and tables 

With these four tasks complete, wedl be in a position to amend the vsearch4web . 
py code to log to MySQL as opposed to a text file. We’ll then use SQL to ask and— 
with luck—answer our questions. 



tJiereiare no 

Dumb Questions 



Do we have to use MySQL here? 


If you want to follow along with the examples in the remainder 
of this chapter, the answer is yes. 



Can I use MariaDB instead of MySQL? 


Yes. As MariaDB is a clone of MySQL, we have no issue with 
you using MariaDB as your database system instead of the “official” 
MySQL. (In fact, over at Head First Labs, MariaDB is a favorite 
among the DevOps team.) 



What about PostgreSQL? Can I use that? 


Emm, eh...yes, subject to the following caveat: if you are 
already using PostgreSQL (or any other SQL-based database 
management system), you can try using it in place of MySQL. 
However, note that this chapter doesn’t provide any specific 
instructions related to PostgreSQL (or anything else), so you may 
have to experiment on your own when something we show you 
working with MySQL doesn’t work in quite the same way with your 
chosen database. There’s also the standalone, single-user SQLite, 
which comes with Python and lets you work with SQL without the 
need for a separate server. That said, which database technology 
you use very much depends on what you’re trying to do. 
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Task 1: Install the MySQL Server 

If you already have MySQL installed on your computer, feel free to move on 
to Task 2. 

How you go about installing MySQL depends on the operating system you’re 
using. Thankfully, the folks behind MySQL (and its close cousin, MariaDB) 
do a great job of making the installation process straightforward. 


□ 

Install MySQL on 

your computer. 

□ 

Install a MySQL 

Python driver. 

n 

Create the database 

i_i 

and tables. 

n 

Create code to 

i_i 

read/write data. 


If you’re running Linux , you should have no trouble finding mysql-server 
(or mariadb-server) in your Software repositories. Use your Software 
installation utility (apt, aptitude, rpm, yum, or whatever) to install 
MySQL as you would any other package. 

If you’re running Mac OS X, we recommend installing Homebrew (find out 
about Homebrew here: http://brew.sh ), then using it to install MariaDB, as in 
our experience this combination works well. 

For all other Systems (including all the various Windows versions), we 
recommend you install the Community Edition of the MySQL server, 
available from: 

http://de v. mysql. com/dozvnloads/mysql/ 

Or, if you want to go with MariaDB, check out: 

https://mariadb. org/dozvnlocid/ 

Be sure to read the installation documentation associated with whichever 
version of the server your download and install. ✓— 

This is going to be ] 
painful, as I've never \ 
used MySQL before... J 


Don’t worry if this is new to you. 

We don’t expect you to be a MySQL whiz-kid 
while working through this material. We’ll 
provide you with everything you need in order 
to get each of our examples to work (even if 
you Ve never used MySQL before). 

If you want to take some time to learn more, 
we recommended Lynn Beighley 3 s excellent 
Head First SQL as a wonderful primer. 



/ 

We II eheek off each 
p*»ple-fced -task as we 
through them. 


0\ all the /VJySQL 
books...i h all the 
world..-this is the ohe 
we broupht to the 
-eh...o++ide whe« we 
■PiTst leav-ned My£QL. 



frientfly 


Head First 

SQL 




QRfUV- 


Li.-i -rLfi 




1 

Ithou^h this is a book 
bout the SQL. <yery 
jnftway, it wses the 
/lySQL- database 
«ara^ement system tor 
ill its examples. Pespite 
ts a^e, it's a stili yeat 

Ie3v-y\’m^ vcsouv-dc. 
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python and mysql 


Iwtroduciwg Pytho^s PP-API 

With the database server installed, let 5 s park it for a bit, while we add support for 
working with MySQL into Python. 

Out of the box, the Python interpreter comes with some support for working 
with databases, but nothing specific to MySQL. What’s provided is a Standard 
database API (application programmer interface) for working with SQL-based 
databases, known as DB-API. What’s missing is the driver to connect the DB- 
API up to the actual database technology you’re using. 

The convention is that programmers use the DB-API when interacting with any 
underlying database using Python, no matter what that database technology 
happens to be. They do that because the driver shields programmers from 
having to understand the nitty-gritty details of interacting with the database’s 
actual API, as the DB-API provides an abstract layer between the two. The 
idea is that, by programming to the DB-API, you can replace the underlying 
database technology as needed without having to throw away any existing code. 

WeTl have more to say about the DB-API later in this chapter. Here’s a 
visualization of what happens when you use Python’s DB-API: 


0 

Install MySQL on 

your computer. 

□ 

Install a MySQL 

Python driver. 

n 

Create the database 

i_i 

and tables. 

n 

Create code to 

1 1_i 

read/write data. 



PythorVs DB-API is defined 
in PEP 0247. That said, don't 
feel the need to run off and 
read this PEP, as it's primarily 
designed to be used as a 
specification by database driver 
implementers (as opposed to 
being a how-to tutorial). 


You wirrtc 'the 
Pytho» £ode Y°* 
need» whith-*- 




Your code 


...uses *the 

stdndav-d Python 

DB-API) whifch" 






PythoiVs 

DB-API 


v/i-th the 

pv-ovided database 
dv-ivev-, whi£h... 




The MySQL- 
Connector/Python 
Driver 


—talks to the 
uhdevlyihg database 
te^hholoav. 




MySQL 


Some programmers look at this diagram and conclude that using Python 5 s DB-API 
must be hugely inefficient. After ali, there are two layers of technology between your 
code and the underlying database System. However, using the DB-API allows you to 
swap out the underlying database as needed, avoiding any database “lock-in,” which 
occurs when you code directly to a database. When you also consider that no two SQL 
dialects are the same, using DB-API helps by providing a higher level of abstraction. 
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Task 2 : Install a MySQL Patabase Priver for 
Python 

Anyone is free to write a database driver (and many people do), but it is typical for 
each database manufacturer to provide an ojficial driver for each of the programming 
languages they support. Oracle , the owner of the MySQL technologies, provides the 
MySQL-Connector/Python driver, and that’s what we propose to use in this chapter. 

There 5 sjust one problem: MySQL-Connector/Python caift be installed with pip. 

Does that mean we’re out of luck when it comes to usin <g MySQL-Connector /Python 
with Python? No, far from it. The fact that a third-party module doesn’t use the 
pip machinery is rarely a show-stopper. All we need to do is install the module “by 
hand’ ? —it 5 s a small amount of extra work (over using pip), but not much. 

Let 5 s install the MySQL-Connector/Python driver by hand (bearing in mind there are other 
drivers available, such as PyMySQL ; that said, we prefer MySQL-Connector/Python, as it’s 
the officially supported driver provided by the makers of MySQL). 

Begin by visiting the MySQL-Connector/Python download page: https://dev.mysql.com/ 
downloads/connector/python/. Landing on this web page will likely preselect your 
operating system from the Select Platform drop-down menu. Ignore this, and adjust the 
selection drop-down to read Platform Independent , as shown here: 


0 

Install MySQL on 

your computer. 

□ 

Install a MySQL 

Python driver. 

^ n 

Create the database 

i_i 

and tables. 

n 

Create code to 

i_i 

read/write data. 


Generally Available (GA) Releases 


Connector/Python 2.1.3 


Select Platform: 

Platform Independent 



Chdhje this 
•field io 

"Platforr* 

/ude pendeh-fc" 


Looking for previous GA versions? 


Platform Independent (Architecture Independent), Compressed TAR 

Archive 

Python 

(mysql-connector-python-2.1.3.tar.gz) 

Platform Independent (Architecture Independent), ZIP Archive 
Python 

(mysql-connector-python-2.1. 3.zip) 


2.1.3 



265.6K 


Download 


MD5: 20bf 8e52e24804915f 9d85claal61c55 | Signature 


347.9K 


Download 


MD5: 710479afc4f 7895207c8f96f91eb5385 | Signature 


We suggest that you use the MD5 checksums and GnuPG signatures to verify the integriXof the packages you download. 


Then, go ahead and click either of the Download buttons (typically, Windows users 
should download the ZIP file, whereas Linux and Mac OS X users can download the 
GZ file). Save the downloaded file to your computer, then double-click on the file to 
expand it within your download location. 


Poni v**nr'/ 'f '1 <Mr 
vevsion IS diffc^Cwt -vrow 
ours ; as lon$ 3s 'rt ' s 3t. 
least vevsion, all is OX 


you are here * 
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install that driver 


Install MySQL-Cowwcctor/Pythow 

With the driver downloaded and expanded on your computer, open a terminal 
window in the newly created folder (if you’re on Windows , open the terminal window 
with Run as Administrator). 

On our computer, the created folder is called mysql-connector-python-2.1.3 
and was expanded in our Downloads folder. To install the driver into Windows , issue 
this command from within the mysql-connector-python-2.1.3 folder: 

py -3 setup.py install 

On Linux or Mac OS X, use this command instead: 

sudo -H python3 setup.py install 

No matter which operating system you’re using, issuing either of the above commands 
results in a collection of messages appearing on screen, which should look similar to 
these: 

running install 
Not Installing C Extension 
running build 
running build_jpy 
running install_lib 
running install_egg_info 
Removing /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/ 
mysql_connector_python-2.1.3-py3.5.egg-info 

Writing /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/ 
mysql_connector_python-2.1.3-py3.5.egg-info 

When you install a module with pip, it runs though this same process, but hides these 
messages from you. What you’re seeing here is the status messages that indicate that 
the installation is proceeding smoothly. If something goes wrong, the resulting error 
message should provide enough information to resolve the problem. If all goes well 
with the installation, the appearance of these messages is confirmation that MySQL- 
Connector/Python is ready to be used. 


0 

Install MySQL on 

your computer. 

□ 

Install a MySQL 

Python driver. 

n 

Create the database 

i_i 

and tables. 

n 

Create code to 

i_i 

read/write data. 



These paths «ay be 
di-Meireivt ov\ your 
to rwpu-fcer. VoCi wov-ry 
abou-fc i-t i-f ih e y av - e . 


tJiereiftre no 

Dumb Questipns 



Should I worry about that “Not Installing C Extension” message? 


No. Third-party modules sometimes include embedded C code, which can help improve computationally intensive Processing. However, 
not all operating systems come with a preinstalled C compiler, so you have to specifically ask for the C extension support to be enabled when 
installing a module (should you decide you need it). When you don’t ask, the third-party module installation machinery uses (potentially slower) 
Python code in place of the C code. This allows the module to work on any platform, regardless of the existence of a C compiler. When a third- 
party module uses Python code exclusively, it is referred to as being written in “pure Python.” In the example above, we’ve installed the pure 
Python version of the MySQL-Connector/Python driver. 
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Task 3 : Create Our Webapp'$ Patabase and Tables 

You now have the MySQL database server and the MySQL-Connector/Python driver installed 
on your computer. It’s time for Task 3, which involves creating the database and the tables 
required by our webapp. 

To do this, you’re going to interact with the MySQL server using its command-line tool, 
which is a small utility that you start from your terminal window. This tool is known as the 
MySQL console. Here’s the command to start the console, logging in as the MySQL database 
administrator (which uses the root user ID): 


0 

Install MySQL on 

your computer. 

0 

Install a MySQL 

Python driver. 

n 

Create the database 

i_i 

and tables. 

n 

Create code to 

i_i 

read/write data. 


mysql -u root -p 

If you set an administrator password when you installed the MySQL server, type in that 
password after pressing the Enter key. Alternatively, if you have no password, just press 
the Enter key twice. Either way, you’11 be taken to the console prompt, which looks like 
this (on the left) when using MySQL, or like this (on the right) when using MariaDB: 


mysql> 


MariaDB [None]> 


Any commands you type at the console prompt are delivered to the MySQL server for 
execution. Let’s start by creating a database for our webapp. Remember: we want to use 
the database to sto re logging data, so the database’s name should reflect this purpose. 
Let’s call our database vsearchlogDB. Here’s the console command that creates our 
database: 


mysql> create database vsearchlogDB; 

The console responds with a (rather cryptic) status message: Query OK, 1 row 
affected (0.00 sec). This is the console’s way of letting you know that 
everything is golden. 

Let’s create a database user ID and password specifically for our webapp to use when 
interacting with MySQL as opposed to using the root user ID all the time (which 
is regarded as bad practice). This next command creates a new MySQL user called 
vsearch, uses “vsearchpasswd” as the new user’s password, and gives the vsearch 
user full rights to the vsearchlogDB database: 



Be swre io -t e v-r«j^a-te 

£o\rr\ft\dy\d you 
Chtcv* iirfco 


WySQL dedole wiih 
5 scmi^oloh. 



mysql> grant all on vsearchlogDB.* to 'vsearch' identified by 'vsearchpasswd'; 


A similar Query OK status message should appear, which confirms the creation of this 
user. Let’s now log out of the console using this command: 

mysql> quit 

You’11 see a friendly Bye message from the console before being returned to your 
operating system. 


1 

Yoia da* we a dittevent 

passwovd it yow 1'ke- Oust 
veroewbev -to youv-s 
as o^osed “to ouvs m "tbc 
e%a**\>les -that -Pollovi- 
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a log table 


Pecide ow a Structure for Your log Pata 

Now that youVe created a database to use with your webapp, you can create any 
number of tables within that database (as required by your application). For our 
purposes, a single table will suffice here, as all we need to store is the data relating to 
each logged web request. 

Recall how we stored this data in a text file in the previous chapter, with each line in 
the vsearch.log file conforming to a specific format: 

•••as well as -the value 
u letters” 

We I09 value ot 
the yhvase"- 



0 

Install MySQL on 

your computer. 

0 

Install a MySQL 

Python driver. 

n 

Create the database 

i_i 

and tables. 

n 

Create code to 

i_i 

read/write data. 


The IP addvess ot the 
tomf wtev that swbmitted the 
data also lo<$ed. 


ImmutableMultiDict([ ('phrase ', 'galaxy'), ( 'letters ', 'xyz')]) |127.0.0.1|Mozilla/5.0 (Macintosh; Intel 

Mac OS X 10 11 2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36|{'x', 1 y 1 } 


f 

The»re s a Jvathev lavge) stvihj that 
deseubes the web bvowsev bemg used. 


Last-bwt not least-the attual 
vesults yvoduted bv seavdhiW 
"lettevs" 'm "fhvase ’ ave also lo<$ed- 


At the very least, the table you create needs five fields: for the phrase, letters, IP 
address, browser string, and resuit values. But let’s also include two other fields: a 
unique ID for each logged request, as well as a timestamp that records when the 
request was logged. As these two latter fields are so common, MySQL provides an 
easy way to add this data to each logged request, as shown at the bottom of this page. 


You can specify the structure of the table you want to create within the console. 
Before doing so, however, let’s log in as our newly created vsearch user using this 
command (and supplying the correct password after pressing the Enter key): 


mysql -u vsearch -p vsearchlogDB ^- 

Here 5 s the SQL statement we used to create the required table (called log). Note 
that the -> Symbol is not part of the SQL statement, as it’s added automatically by 
the console to indicate that it expects more input from you (when your SQL runs to 
multiple lines). The statement ends (and exeeutes) when you type the terminating 
semicolon character, and then press the Enter key: 

mysql> create table log ( 


This is *the 
donsole s 
Continuatio* 
symbol 




-> id int auto_increment primary key, 

-> ts timestamp default current_timestamp, 
-> phrase varchar(128) not null, 

-> letters varchar(32) not null, 

-> ip varchar(16) not null, 

-> browser_string varchar(256) not null, 

-> resuits varchar(64) not null ); 


Remembev-: we 
set this usev s 
passwov-d to 
Nseavchpasswd • 


/V!y£#L will autor»atidally 
pvovide data &v these 
tields. 


These -fields will hold the 
data -fov each ve<\uest (as 
pv-ovided in the -fov-m data). 
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Cowfirw Your Table Is Ready for Pata 

With the table created, we’re done with Task 3. 

Leti confirm at the console that the table has indeed been created with the 
structure we require. While stili logged into the MySQL console as user 
vsear ch, issue the describe log command at the prompt: 


0 

Install MySQL on 

your computer. 

0 

Install a MySQL 

Python driver. 
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Create the database 

UiJ 
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mysql> describe log; 


+ - 

| Field 


.+ - 

| Type 


.+ - 

| Extra 


■+ 

I 


| Null | Key | Default 


+- 

-+■ 


_+- 

- 

-+■ 


_+- 

-+ 

1 id 

i 

int(11) 

| NO 

| PRI 

i 

NULL 

| auto increment 

i 

| ts 

i 

timestamp 

| NO 

i 

i 

CURRENT_T IME S T AMP 

i 

i 

| phrase 

i 

varchar(128) 

| NO 

i 

i 

NULL 

i 

i 

| letters 

i 

varchar(32) 

| NO 

i 

i 

NULL 

i 

i 

1 ip 

i 

varchar(16) 

| NO 

i 

i 

NULL 

i 

i 

| browser string 

i 

varchar(256) 

| NO 

i 

i 

NULL 

i 

i 

| results 

i 

varchar(64) 

| NO 

i 

i 

NULL 

i 

i 

+- 

-+■ 


_+- 

- 

-+■ 


_+- 

-+ 


And there it is: proof that the log table exists and has a structure that fits with our 
web applicationi logging needs. Type qui t to exit the console (as you are done 
with it for now). 



So now I'm ready to add data to 
the table, right? My friend who's an 
SQL expert says I can use a bunch of 
manual INSERT statements to do that 


Yes, that’s one possibility. 

There i nothing stopping you from monually 
typing a bunch of SQL INSERT statements 
into the console to monually add data to your 
newly created table. But remember: we want 
our webapp to add our web request data to the 
log table automatically, and this applies to 
INSERT statements, too. 


To do this, we need to write some Python code 
to interact with the log table. And to do that , 
we need to learn more about Pythoni DB-API. 


you are here ► 
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python’s db-api 



X^B-APIlIp Clo.se, 1 op s 


Recall the diagram from earlier in this chapter that positioned Python’s DB-API in relation to 
your code, your chosen database driver, and your underlying database System: 


Your code 



Pythorfs 

DB-API 




MySQL 


Connector/Python 

Driver 


The promise of using DB-API is that you can replace the driver/database combination with 
very minor modifications to your Python code, so long as you limit yourself to only using the 
facilities provided by the DB-API. 

Let 5 s review what’s involved in programming to this important Python Standard. We are going 
to present six steps here. 

W-API Step 1: Pefiwe your cowwectiow charactcristics 

There are four pieces of information you need when connecting to MySQL: (1) the IP address/name of the 
computer running the MySQL server (known as the host ), (2) the user ID to use, (3) the password associated 
with the user ID, and (4) the name of the database the user ID wants to interact with. 


The MySQL- Connector/Python driver allows you to put these connection characteristics into a Python dictionary 
for ease of use and ease of reference. Let 5 s do that now by typing the code in this Up Close into the >>> prompt. 
Be sure to follow along on your computer. Here 5 s a dictionary (called dbconf ig) that associates the four 
required “connection keys” with their corresponding values: 


|. OiaV sevvev- is ov\ ouv- 


loddl 


addvess *Cov w host • 

»> dbconfig = 



2- The "vseairdh" use»- (£> 

+to«, eairlieir i» -this ehaptev- is 
Sssigned -to -the "user" key. 



host 

user 


127.0.0.1 
vsearch’, 


Z. The yassword” 
key ,s assi<y\ed “the 
dov-redt password 
{fi use with our 
user IP* 


'password' 
'database' 


'vsearchpasswd', 


'vsearchlogDB', } 

The database «a^e-Veav-thlo^PB^ 
th',s tase—is assi^ed to the database y 
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PP-API Step 2 : Import your database driver 

With the connection characteristics defined, it’s time to import our database driver: 


»> import mysql. connector 

This import makes the MySQL-specific driver available to the DB-API. 


“the dvWev- 
-£ov dd*tdbasc 
you av-C 


PILAP1 Step $: Establish a connection to the server 

Let’s establish a connection to the server by using the DB-APLs connect function to establish our 
connection. Let’s save a reference to the connection in a variable called conn. Here’s the call to connect, 
which establishes the connection to the MySQL database server (and creates conn): 


»> conn = mysql. connector. connect(**dbconfig) 

v 


This tali establishes the tovmetW 



Pass in -the did*tionav"y o-f 
£onne£-fcion ^hdra^fcem-fcids. 


Note the strange ** that precedes the single argument to the connect function. (If you 5 re a G/G++ 
programmer, do not read * * as “a pointer to a pointer,” as Python has no notion of pointers.) The * * 
notation telis the connect function that a dictionary of arguments is being supplied in a single variable (in 
this case dbconf ig, the dictionary you just created). On seeing the **, the connect function expands the 
single dictionary argument into four individual arguments, which are then used within the connect function 
to establish the connection. (You’11 see more of the ** notation in a later chapter; for now, just use it as is.) 


PP-API Step 4: Open a cursor 

To send SQL commands to your database (via the just-opened connection) as well as receive results from your 
database, you need a cursor. Think of a cursor as the database equivalent of the file handle from the last chapter 
(which lets you communicate with a disk file once it was opened). 

Greating a cursor is straightforward: you do so by calling the cursor method included with every connection 
object. As with the connection above, we save a reference to the created cursor in a variable (which, in a wild fit 
of imaginative creativity, weVe named cursor): 

»> cursor = conn. cursor () <= -Cvea*te a fcuv-sov “to send 

to»»av»ds -to -tbe sevvev, a*d to 

veteive results- 

We are now ready to send SQL commands to the server, and—hopefully—get some results back. 

But, before we do that, let’s take a moment to review the steps completed so far. WeVe defined the connection 
characteristics for the database, imported the driver module, created a connection object, and created a 
cursor. No matter which database you use, these steps are common to all interactions with MySQL (only the 
connection characteristics change). Keep this in mind as you interact with your data through the cursor. 
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DB-APlXJpCkse,2rf3 - 

With the cursor created and assigned to a variable, it 5 s time to interact with the data in your 
database using the SQL query language. 


P&-API Stcp 5: Po the SQL thiwg! 


The cursor variable lets you send SQL queries to MySQL, as well as retrieve any results produced by 
MySQL’s processing of the query 

As a general rule, the Python programmers over at Head First Labs like to code the SQL they intend to send to 
the database server in a triple-quoted string, then assign the string to a variable called _SQL. A triple-quoted 
string is used because SQL queries can often run to multiple lines, and using a triple-quoted string temporarily 
switches off the Python interpretehs “end-of-line is the end-of-statement” rule. Using _SQL as the variable 
name is a convention among the Head First Labs programmers for defining constant values in Python, but you 
can use any variable name (and it doesn’t have to be all uppercase, nor prefixed within an underscore). 


Let’s start by asking MySQL for the names of the tables in the database we’re connected to. To do this, assign 
the show tables query to the _SQL variable, and then call the cursor . execute function, passing _SQL 
as an argument: 


Assior the »> _SQL = " " " show tables" " " 

SQL. <\uery >» cursor.execute(_SQL) 

{p a variable* 


Servd the «\uery 'm the 
"jSQL" variable to MySQL- 

•for e*eiution. 


When you type the above cursor, execute command at the >>> prompt, the SQL query is sent to your 
MySQL server, which proceeds to execute the query (assuming it 5 s valid and correct SQL). However, any 
results from the query dordt appear immediately; you have to ask for them. 

You can ask for results using one of three cursor methods: 

• cursor . fetehone retrieves a single row of results. 

• cursor . fetchmany retrieves the number of rows you specify. 

• cursor . fetchall retrieves all the rows that make up the results. 


For now, let’s use the cursor . fetchall method to retrieve all the results from the above query, assigning 
the results to a variable called res, then displaying the contents of res at the >>> prompt: 


£jet all tbe 

data returned 

W M7SQL. 


»> res = cursor. fetchall () 

»> res __ 

[ (' log ' ,) ] ~~- Display tbe results- 


The contents of res look a little weird, don’t they? You were probably expecting to see a single word here, 
as we know from earlier that our database (vsearchlogDB) contains a single table called log. However, 
what’s returned by cursor . fetchall is always a list of tuples , even when there’s only a single piece of data 
returned (as is the case above). Let 5 s look at another example that returns more data from MySQL. 
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Our next query, describe log, queries for the information about the log table as stored in the database. 
As you’11 see below, the information is shown twice: once in its raw form (which is a little messy) and then over 
multiple lines. Recall that the resuit returned bycursor.fetchallisa list of tuples. 


Here 5 s cursor . f etchall in action once more: 


|t \ooVs a 
I\ttI* wCSS'/, 

wi rw.s 

\s a Vist 

U^ es ' 


Take SQL. nu ^y... 

“ .theh send it to the sev-vev... 
...and iie» atdess ihe resulis. 


»> _SQL = """describe log""" 

»> cursor. exeeute (_SQL) 

»> res = cursor. f etchall () 

»> res 

[('id' , 'int(11) ', 'NO 1 , 'PRI', None, 'auto_increment ') , ('ts' , 

' timestamp' , 'NO A ' ' , ' CURRENT_TIMESTAMP' , ' ') f ('phrase ' , 

'varchar(128 )', 'NO', '', None, ''), ('letters', 'varchar(32)', 

'NO', '', None, '’), ('ip', 'varchar(16)', 'NO', '', None, ’’), 

('browser_string', 'varchar(256)', 'NO', '', None, ’’), ('resuits', 

'varchar(64)', 'NO', '', None, ’')] 


Eaeh tuplc 
-Pirorh the 
list ot 
iuplcs is 
oh its 
owh lihc. 


»> for row in res: 
print(row) 




Tjke each row i h ih e replis- 
a»d aisylay vt iis ow* Ime- 



('id', 'int(11)', 'NO', 'PRI', None, 'auto_increment') 

('ts', 'timestamp', 'NO', '', 'CURRENT_TIMESTAMP', '') 
('phrase', 'varchar(128)', 'NO', '', None, '') 
('letters', 'varchar(32)', 'NO', '', None, '') 

('ip', 'varchar(16)', 'NO', '', None, '') 

('browser_string', 'varchar(256)', 'NO', '', None, '') 

('results', 'varchar(64)', 'NO', '', None, '') 


The per-row display above may not look like much of an improvement over the raw output, but compare it to 
the output displayed by the MySQL console from earlier (shown below). What 5 s shown above is the same data 
as what’s shown below, only now the data is in a Python data structure called res: 


mysql> describe log; 
+ 



Field 

1 

Type 

1 

Null 

1 

Key 

1 

Default 

_+- 

| Extra 

id 

T 

1 

int(11) 

i 

1 

NO 

T 

1 

PRI 

T 

1 

NULL 

~r 

| auto increment 

ts 

1 

timestamp 

1 

NO 

1 


1 

CURRENT_TIMESTAMP 

1 

phrase 

1 

varchar(128) 

1 

NO 

1 


1 

NULL 

1 

letters 

1 

varchar(32) 

1 

NO 

1 


1 

NULL 

1 

ip 

1 

varchar(16) 

1 

NO 

1 


1 

NULL 

1 

browser string 

1 

varchar(256) 

1 

NO 

1 


1 

NULL 

1 

results 

1 

varchar(64) 

1 

NO 

1 


1 

NULL 

1 










_+- 


+ 

I 

+ 


t-ook dosely. 

lis ihe 
data. 
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even more db-api 



I®'API tip CloSe, 3 o?3 - 

Lets use an insert query to add some sample data to the log table. 


Its tempting to assign the query shown below (which weVe written over multiple lines) to the _SQL 
variable, then call cursor . execute to send the query to the server: 

»> _SQL = """insert into log 

(phrase, letters, ip, browser_string, resuits) 
values 

('hitch-hiker', 'aeiou', '127.0.0.1’, 'Firefox', "{'e', 'i'}")""" 


»> cursor. execute (_SQL) 

Dofft get us wrong, whats shown above does work. However, hardcoding the data values in this way is rarely 
what you’11 want to do, as the data values you sto re in your table will likely change with every insert. 
Remember: you plan to log the details of each web request to the log table, which means these data values 
will change with every request, so hardcoding the data in this way would be a disaster. 

To avoid the need to hardcode data (as shown above), Pythoffs DB-API lets you position “data placeholders” in 
your query string, which are filled in with the actual values when you call cursor, execute. In effect, this 
lets you reuse a query with many different data values, passing the values as arguments to the query just before 
its executed. The placeholders in your query are stringed values, and are identified as %s in the code below. 


Compare these commands below with those shown above: 


»> SQL = 


"""insert into log 

(phrase, letters, ip, browser_string, resuits) 
values 


( 


*s. 


9- es 
S 


9- o 
^5 S 


9-o 
T5 S 


a 

"o 


s)" 


u h 




Whch domposma 
y°u(r y, use DB 
API pbdeholdev-s 
insiead ot actual 
data values. 



»> cursor.execute( SQL, ('hitch-hiker', 'xyz', '127.0.0.1', 'Safari', 'set()')) 


There are two things to note above. First, instead of hardcoding the actual data values in the SQL query, we 
used the % s placeholder, which telis DB-API to expect a stringed value to be substituted into the query prior 
to execution. As you can see, there are five %s placeholders above, so the second thing to note is that cursor . 
execute call is going to expect five additional parameters when called. The only problem is that cursor . 
execute doesfft accept just any number of parameters; it accepts at most two. 

How can this be? 

Looking at the last line of code shown above, its ciear that cursor . execute accepts the five data values 
provided to it (without complaint), so what gives? 

Take another, closer look at that line of code. See the pair of parentheses around the data values? The use of 
parentheses turns the five data values into a single tuple (containing the individual data values). In effect, the 
above line of code supplies two arguments to cursor, execute: the placeholder-containing query, as well as 
a single tuple of data values. 

So, when the code on this page executes, data values are inserted into the log table, right? WelL.not quite. 
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When you use cursor . execute to send data to a database system (using the insert query), the data may 
not be saved to the database immediately. This is because writing to a database is an expensive operation 
(from a processing-cycle perspective), so many database Systems cache inserts, then apply them all at once 
later. This can sometimes mean the data you think is in your table isn’t there yet, which can lead to problems. 

For instance, if you use insert to send data to a table, then immediately use select to read it back, the 
data may not be available, as it is stili in the database systenfs cache waiting to be written. If this happens, 
you’re out of luck, as the select fails to return any data. Eventually, the data is written, so ifs not lost, but 
this default caching behavior may not be what you desire. 


If you are happy to take the performance hit associated with a database write, you can force your database 
system to commit all potentially cached data to your table using the conn . commit method. Lefs do that now 
to ensure the two insert statements from the previous page are applied to the log table. With your data 
written, you can now use a select query to confirm the data values are saved: 


"Forte” a*y tacMed —»> 

io be wH-t-ken »> 

■fco -the -table. »> 

»> 

ttevVs ibe w id 
value 

auWa-btally 

assi<vned “to ibis (1 , 

(2 

VO>W... ' ' 



conn.commit() 

_SQL = """select * from log""" 
cursor.execute(_SQL) 
for row in cursor.fetchall(): 
print(row) 


date time. date time (2016, 3, . . . , " { ’ e' , ' i ' } ") j vc ^bv-idyd ibe 

datetime.datetime(2016, 3, . . . , ’ set()') oui? , 3 

Jfvt oy\ ibis pa^e- 

— -a*d here s wb ai ii iilled in V' (Wiamp). 


Rei\r ieve tbe 

just—wmitteh 

data. 


From the above you can see that MySQL has automatically determined the correct values to use for id and 
ts when data is inserted into a row. The data returned from the database server is (as before) a list of tuples. 
Rather than save the results of cursor . fetchall to a variable that is then iterated over, weVe used 
cursor . fetchall directly in a for loop in this code. Also, don’t forget: a tuple is an immutable list and, as 
such, supports the usual square bracket access notation. This means you can index into the row variable used 
within the above for loop to pick out individual data items as needed. For instance, row [ 2 ] picks out the 
phrase, row [ 3 ] picks out the letters, and row [ -1 ] picks out the results. 


PP-API Stcp 6: Close your cursor and connection 


With your data committed to its table, tidy up after yourself by closing the cursor as well as the connection: 


»> cursor. close () 
True 

»> conn. close () 



/is alv/ays a 

3°°d idea io 
iidy up. 


Note that the cursor conflrms successful closure by returning True, while the connection simply shuts 
down. If s always a good idea to close your cursor and your connection when they 5 re no longer needed, as 
your database system has a finite set of resources. Over at Head First Labs , the programmers like to keep their 
database cursors and connections open for as long as required, but no longer. 
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the tasks are done 


Task 4: Create Code to Work with Our 
Webapp s Patabase and Tables 

With the six steps of the DB-API Up Close completed, you now have the code needed to 
interact with the log table, which means you Ve completed Task 4: Create code to work 
with our webapps database and tables. 

Let 5 s review the code you can use (in its entirety): 



Pe-f me youv 

dovmedtio* 

dhav-adtev-istids. 


dbconfig = { 'host' : '127.0.0.1', 

'user': 'vsearch', 

'password*: *vsearchpasswd’, 

'database': *vsearchlogDB', } 

import mysql.connector 



0 

Install MySQL on 

your computer. 

0 

Install a MySQL 

Python driver. 

0 

Create the database 

and tables. 

0 

Create code to 

read/write data. 

V_ 0uv task list 
is done! 


the database dvivev. 


conn = mysql. connector. connect (**dbconfig) 

dorwedtion ^ 

a^ _9 cursor = conn. cursor () 

a duVSOV". 

_SQL = """insert into log 

(phrase, letters, ip, browser_string, results) ^ 
values 

( O, _ Q__ O, _ Q__ O, _ \ I! I! I! 

■oS, •bS , •bS, '6SJ 


Assi op a <\uevy to a 
stvrn^ (*>ote tV>e Vwe 

platekoldev av^rxents)- 



cursor.exeeute( SQL, (*galaxy', *xyz', '127.0.0.1*, 'Opera', "{'x', 'y'}")) 


r 


fbvde the 
database 
to wvitd 
youv data* 


conn.commit() 

_SQL = """select * from log""" 
cursor.exeeute( SQL) 


for row in cursor.fetchall(): 
print(row) 



cursor.close() 
conn.close() 


t 

airguiwehk (i* a twple). " 


Tidy up when you Ve 
dohe* 


Retvieve the (just wvittcv>) 
data -firow» tbe table, 
dis?laym^ tbe outf ut ^ 

by v-ow* 


With each of the four tasks now complete, you 5 re ready to adjust your webapp to log 
the web request data to your MySQL database system as opposed to a text file (as is 
currently the case). Let 5 s start doing this now. 
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DataBase Magnets 


Take another look at the log_request function from the last chapter. 

Recall that this small function accepts two arguments: a web request object, and the 
results of the vsearch: 


def log request(req: 'flask_request', res: str) -> None: 
with open('vsearch.log', 'a') as log: 

print(req.form, req.remote addr, req.user agent, res, file=log, 


sep=' 


) 


Your job is to replace this functioni suite with code that logs to your database (as 
opposed to the text file).The def line is to remain unchanged. Decide on the magnets 
you need from those scattered at the bottom on this page, then position them to 
provide the functioni code: 


def log request(req: 'flask request', res: str) -> None: 


Cvikey/ n/ha-t 
d «^CSS O-P 

•^aghc-ts. Ca* 
you Help? 



cursor = conn.cursor() 



SQL = """select * from log""" 


cursor. execute (_JSQL, 


(req.form['phrase'], 
req.form['letters '] r 

req.remote_addr, 

req.user_agent.browser 
res, )) 



dbconfig = { 'host': '127.0.0.1', 

'user': 'vsearch', 

'password': 'vsearchpasswd', 

'database': 'vsearchlogDB', } 


SQL = """insert into log 

(phrase, letters, ip, browser_string, results) 

values 


(*6S f *6S , oS , oS , oS) 


for row in cursor.fetchall0 
print(row) 
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log to mysql 


\ 


k 


DataBase Magnets Solution 

You were to take another look at the log_request function from the last chapter: 

def log request(req: 'flask_request' , res: str) -> None: 

with open('vsearch.log' , 'a') as log: 

print(req.form, req.remote addr, req.user agent, res, file=log, sep= 

Your job was to replace this functiorYs suite with code that logs to your database.The 
def line was to remain unchanged. You were to decide which magnets you needed from 
those scattered at the bottom on the page. 


i i f 


def log request(req: 'flask request', res: str) -> None: 


dbconfig = { 'host': '127.0.0.1', 

'user': 'vsearch', 

'password': 'vsearchpasswd', 

'database': 'vsearchlogDB', } 


import mysql.connector 
conn = mysql.connector.connect(**dbconfig) 
cursor = conn.cursor() j 


De+ine the donnedtion 
£ha\ra£te\risti£s. 


SQL = """insert into log 

(phrase, letters, ip, browser_string, resuits) 

values 

/ o _ Q,— 9 ,— g. c 2- Q \ M M M 

(*6S, oS , oS, oS, oS) 


mpovt "the dvivcv-, the* 
establish a dormedtion, 
a*d the* £v-eate a 
duvsov*. 

Cv-eate a 

dontainin* the 
want 




<\ucv"Y you 

“to USC. 


cursor.exeeute( SQL, 


(req.form[’phrase'], 
req.form['letters'], 
req.remote_addr, 
req.user_agent.browser, 
res, )) 


These mayicts 
wevWt ^ccdcd- 


conn. commit () 


] 


cursor.close() 


] 


conn.close() 


] 



/VCtev- ensvvin^ 

the data is saved, 
vieVe tidyin^ >*? W 
£.los'm$ the duv-sov 

3*d the 


cursor.exeeute(_SQL) 

SQL = """select * from log""" 



Exeeute the ^uevy. 


This is v-athev- thar> stov-e 
the e^tiv-e bv-ov/sev strmj (stov-ed 
m Ve«\-use v-__a<je^ t ,; ) ; v/eVe only 
extv-adtmg the r>a**e o£ the 
b vowsev- 


for row in cursor 
print(row) 


fetchall() 
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Change the code in your vsearch4web. py file to replace the original log_request function’s 
code with that from the last page. When you have saved your code, start up this latest version of your 
webapp at a command prompt. Recall that on Windows, you need to use this command: 

C:\webapps> py -3 vsearch4web.py 


While on Linux or Mac OSX, use this command: 

$ python3 vsearch4web.py 

Your webapp should start running at this web address: 

http://127. 0.0.1:5000/ 


Use your favorite web browser to perform a few searches to confirm that your webapp runs fine. 
There are two points we’d like to make here: 

■ Your webapp performs exactly as it did before: each search returns a “results page” to the 
user. 


Lojj ih to 

the /Wy£$L 
dohsole. 


• Your users have no idea that the search data is now being logged to a database table as 
opposed to a text file. 

Regrettably, you can’t use the /viewlog URL to view these latest log entries, as the function 
associated with that URL (view_the_log) only works with the vsearch. log text file (not 
the database). We’ll have more to say about fixing this over the page. 

For now, let’s conclude this Test Drive by using the MySQL console to confirm that this newest 
version of log_request is logging data to the log table. Open another terminal window and 
follow along (note: we’ve reformatted and abridged our output to make it fit on this page): 

This c^uev-y asks to see all the data m the w log 
table (youv adtual data v/ill likely di£Cev). 


File Edit Window Help Checking our log DB 


$ mysql -u vsearch -p vsearchlogDB 
Enter password: 


Welcome to MySQL monitor... 


mysql> select * from log; 


2016-03-09 13:40:46 
2016-03-09 13:42:07 
2016-03-09 13:42:15 
2016-03-09 13:43:07 



phrase 


life, the uni 
hitch-hiker 
galaxy 
hitch-hiker 



ything 


aeiou 

aeiou 

xyz 

xyz 


127.0.0.1 

127.0.0.1 

127.0.0.1 

127.0.0.1 


browser string 


firefox 

safari 

chrome 

firefox 


{'u', 'e', 'i', 'a'} | 
{'i', 'e'} ! 
{'y', 'x'} ! 
set () ! 
- + 


4 rows in set (0.0 sec) 


mysql> quit 
Bye 


1 


t tovget to <^uit the 
£onsole v/hen you Ve done- 


Re**e**bev-: weVe only stormj the bvov/sev na**e- 
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Storing Pata 1$ Only Half the Pattle 

Having run though the Test Drive on the last page, youVe now confirmed that your 
Python DB-API-compliant code in log_request does indeed store the details of 
each web request in your log table. 

Take a look at the most recent version of the log_request function once more 
(which includes a docstring as its first line of code): 


def log_request (req: ' flask_request' , res: str) -> None: 
"""Log details of the web request and the resuits.""" 
dbconfig = { 'host': '127.0.0.1', 

'user': 'vsearch', 

’password’: 'vsearchpasswd’, 

'database': 'vsearchlogDB', } 


import mysql.connector 


conn = 
cursor 
SQL = 


mysql.connector.connect(**dbconfig) 

= conn.cursor() 

"""insert into log 

(phrase, letters, ip, browser_string, results) 
values 


( o, _ g, _ g, — g, _ g,_\ !inn 

'oS r *6S r o S j 'dS r nSj 

cursor.execute(_SQL, (req.form['phrase'], 

req.form[ 1 letters 1 ], 
req.remote_addr, 
req.user_agent.browser, 
res, )) 

conn.commit() 
cursor.close() 
conn.close() 

This wew fuwctiow is a big chawge 

There 5 s a lot more code in the log_request function now than when it operated on 
a simple text file, but the extra code is needed to interact with MySQL (which youVe 
going to use to answer questions about your logged data at the end of this chapter), so 
this new, bigger, more complex version of log_request appears justified. 

However, recall that your webapp has another function, called view_the_log, 
which retrieves the data from the vsearch.log log file and displays it in a nicely 
formatted web page. We now need to update the view_the_log functioni code to 
retrieve its data from the log table in the database, as opposed to the text file. 



ExperiehCed Pytho* 

well look at this 
■rundtioh’s Code and 
let out a gasp o-f 
disapp(roval. /ou'11 
lea»rn why in a -Pew 
-time. 


The question is: what’s the best way to do this? 
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How Pest to Rcuse Your Patabase Code? 

You now have code that logs the details of each of your webapp’s requests to MySQL. 
It shoukhft be too much work to do something similar in order to retrieve the data 
from the log table for use in the view_the_log function. The question is: what’s 
the best way to do this? We asked three programmers our question...and got three 
different answers. 




O 

0 



Isn't it ciear it's time we 
considered using classes 
and objects as the correct 
way to handle this type of 
reuse? 


O 

o 



In its own way, each of these suggestions is valid, if a little suspect (especially the first 
one). What may come as a surprise is that, in this case, a Python programmer would 
be unlikely to embrace any of these proposed Solutions on their own. 
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Cowsider What YouYe Tryiwg to Reuse 

Let’s take another look our database code in the log_request function. 

It should be ciear that there are parts of this function we can reuse when writing 
additional code that interacts with a database System. Thus, weVe annotated the 
function 5 s code to highlight the parts we think are reusable, as opposed to the parts 
that are specific to the Central idea of what the log_request function actually does: 


def logrequest(req: 'flask_request ', res: str) -> None: 

details of the web recreest and the results.""" 


u u 



These *two 

ave aUays 
<^om^ *to be 
the sa^e, so 
ea y \ be veused- 


These thvee 
s-tatemchts 
ave also 

always tb c ^ 
sa^c, so eah 
be veused, -to, 


import mysql.connector 


The database eoirmeetioh 
^havae-tevis-ties ave vevy 
speei-fie to what weVe 
doihg heve, but ave 
lik «ly »eeded ih othev 
plaees, so should be 
reusable. 



Based on this simple analysis, the log_request function has three groups of code 
statements: 

* statements that can be easily reused (such as the creation of conn and cursor, 
as well as the calls to commi t and closel 


TWis toit is tv»« ' rea! 'V*k 
ot UaU ywft OK 'ms.de the 
U» a«d be veused 

m a*/ meawACJ «*[ (as 
lo b> tbe job at 

ha*d)- 


statements that are specific to the problem but stili need to be reusable (such as the 
use of the dbconf ig dictionary); and 

statements that cannot be reused (such as the assignment to _SQL and the call 
to cursor . exeeute). Any further interactions with MySQL are very likely to 
require a different SQL query, as well as different arguments (if any). 
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What About That Import? 



Nope, we didn’t forget. 

The import mysql. connector statement 
wasrht forgotten when we considered reusing the 
log_request functioni code. 

This omission was deliberate on our part, as 
we wanted to call out this statement for special 
treatment. The problem isn’t that we don’t want to 
reuse that statement; it’s that it shouldnl appear in 
the function’s suite! 


Pe caref ul when positioning your import statements 

We mentioned a few pages back that experienced Python programmers may well look 
at the log_request function’s code and let out a gasp of disapproval. This is due 
to the inclusion of the import mysql. connector line of code in the function’s 
suite. And this disapproval is in spite of the fact that our most recent Test Drive clearly 
demonstrated that this code works. So, what’s the problem? 

The problem has to do with what happens when the interpreter encounters an 
import statement in your code: the imported module is read in full, then executed by 
the interpreter. This behavior is fine when your import statement occurs outside of a 
function , as the imported module is (typically) only read once : then executed once. 

However, when an import statement appears within a function, it is read and executed 
every time the function is called. This is regarded as an extremely wasteful 
practice (even though, as weVe seen, the interpreter won’t stop you from putting an 
import statement in a function). Our advice is simple: think carefully about where 
you position your import statements, and don’t put any inside a function. 
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Cowsidcr What YouYe Tryiwg to Po 


In addition to looking at the code in log_request from a reuse perspective, it 5 s also possible 
to categorize the function’s code based on when it runs. 

The “guts” of the function is the assignment to the _SQL variable and the call to cursor . 
execute. Those two statements most patently represent what the function is meant to do, 
which—to be honest—is the most important bit. The function’s initial statements define 
the connection characteristics (in dbconf ig), then create a connection and cursor. This 
setup code always has to run before the guts of the function. The last three statements in the 
function (the single commit and the two closes) execute after the guts of the function. This is 
teardown code, which performs any required tidying up. 


With this setup\ do , teardown pattern in mind, let’s look at the function once more. Note that weVe 
repositioned the import statement to execute outside of the log_request function 5 s suite 
(so as to avoid any further disapproving gasps): 


import mysql.connector 


This is a better ftiet far a»y import statements 
Ithat IS, outside the Wtio*>'s suite). 


def log_request(req: 'flask_request’, res: str) -> None: 
"""Log details of the web request and the results.""" 



Wouldrft it be neat if there were a way to reuse this setup, do, 
teardown pattern? 
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YouVe Secw This Pattern Pefore 


Gonsider the pattern we just identified: setup code to get ready, followed by code to do 
what needs to be done, and then teardown code to tidy up. It may not be immediately 
obvious, but in the previous chapter, you encountered code that conforms to this 
pattern. Here it is again: 


0pev\ ‘the 'file* 




with open('todos.txt') as tasks 
for chore in tasks: 

print(chore, end='') 


Assijh -the file 

o ci 

vaHablc. 


P&rfo\rm sor*e piro£essi 


*9- 


Recall how the with statement manages the context within which the code in its suite 
runs. When you’re working with files (as in the code above), the with statement 
arranges to open the named file and return a variable representing the file stream. In 
this example, thafs the tasks variable; this is the setup code. The suite associated 
with the with statement is the do code; here thafs the for loop, which does the 
actual work (a.k.a. “the important bit”). Finally, when you use with to open a file, 
it comes with the promise that the open file will be closed when the with’s suite 
terminates. This is the teardown code. 

It would be neat if we could integrate our database programming code into the with 
statement. Ideally, it would be great if we could write code like this, and have the 
with statement take care of all the database setup and teardown details: 



]flle st»U ficed 
*t o dtf me "the 

C.orme£.-boir\ 

dhavafrteristits. 


dbconfig = { 'host': '127.0.0.1', 

'user': 'vsearch', 

'password': 'vsearchpasswd', 

'database': 'vsearchlogDB', } 


Thls stater*e*t 

wo ^ wrth databases 
as °PP<«ed to disk -files, 

a«d vetuv-hs a d UV . Sov . £ 

us io Wotrk wifh. 


o\r 


The u do 
lodt 
“the las-fe 

P a 9 c \rer*aihs 

uhdhahged. 




with UseDatabase(dbconfig) as cursor: 

_SQL = """insert into log 

(phrase, letters, ip, browser_string, resuits) 
values 

( O, _ O, _ O. _ O. _ O. _ \ IT ff IV 

■oS, , '6S , "oS , 'dS) 

cursor.exeeute(_SQL, (req.form['phrase'], 

req.form['letters'], 
req.remote_addr, 
req.user_agent.browser, 
res, )) 



The good news is that Python provides the context management protocol, which 
enables programmers to hook into the with statement as needed. Which brings us to 
the bad news... 



i iry io »- Uh this 
(■ode, as you 'v e yet to 

UseDatabase 
iontext *«anagev. 
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The Pad News IsrTt Really AII That Pad 

At the bottom of the last page, we stated that the good news is that Python provides 
a context management protocol that enables programmers to hook into the with 
statement as and when required. If you learn how to do this, you can then create 
a context manager called UseDatabase, which can be used as part of a with 
statement to talk to your database. 

The idea is that the setup and teardown “boilerplate” code that you Ve just written 
to save your webapp 5 s logging data to a database can be replaced by a single with 
statement that looks like this: 


with UseDatabase(dbconfig) as cursor: 


The bad news is that creating a context manager is complicated by the fact that you need 
to know how to create a Python class in order to successfully hook into the protocol. 

Gonsider that up until this point in this book, you Ve managed to write a lot of usable 
code without having to create a class, which is pretty good going, especially when 
you consider that some programming languages doni let you do anything without first 
creating a class (we’re looking atjozq Java). 


This "wi-th" 
s-fca-fcemen-t is 
siwilav io ih e one 
used wi-fch -files snd 
^he "ope*" B IF, 
e*cepi that this 
°*e works with a 
database instead- 



However, it’s now time to bite the bullet (although, to be honest, creating a class in 
Python is nothing to be scared of). 


As the ability to create a class is generally useful, leti deviate from our current 
discussion about adding database code to our webapp, and dedicate the next (short) 
chapter to classes. Weil be showing you just enough to enable you to create the 
UseDatabase context manager. Once that’s done, in the chapter after that, weil 
return to our database code (and our webapp) and put our newly acquired class-writing 
abilities to work by writing the UseDatabase context manager. 
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Chapter 7'$ Code 


TW.s is tV daUase 

tode W*at e»«H 
6 L Wm vowV 'Me'oa\>? 

VJUA"*' 7 

(u, "lo^r est 


import mysql.connector 

def log_request(req: 'flask_request', res: str) -> None: 
MMM Log details of the web request and the results.""" 

dbconfig = { 'host': '127.0.0.1', 

'user': 'vsearch', 

'password': 'vsearchpasswd', 

'database': 'vsearchlogDB', } 

conn = mysql.connector.connect(**dbconfig) 
cursor = conn.cursor() 

_SQL = """insert into log 

(phrase, letters, ip, browser_string, results) 
values 

/ o. 0 O- o 9- o 9- o 9- o \ f| f| f| 

^ O O f o o ^ o o ^ o o ^ oo^ 

cursor.execute(_SQL, (req.form['phrase'], 

req.form['letters'], 
req.remote_addr, 
req.user agent.browser, 
res, )) 

conn.commit () 
cursor.close () 
conn.close () 


dbconfig = { 'host': '127.0.0.1', 

'user': 'vsearch', 

'password': 'vsearchpasswd', 

'database': 'vsearchlogDB', } 

with UseDatabase(dbconfig) as cursor: 

_SQL = """insert into log 

(phrase, letters, ip, browser_string, results) 
values 

/9-0 9-0 9-0 9-0 9- O \ !!!!!! 

O O /• OOf O O /- o o ^ 

cursor.execute(_SQL, (req.form['phrase'], 

req.form['letters'], 
req.remote_addr, 
req.user agent.browser, 
res, )) 



to do the sa we 
thimg as owr eunrent Code 
^pla^g the iie ih ihe 

Hjre^uest WtioU. 

Bui do»‘l tv*y io r Uh £ hjs 
to f e yei, as ii Wfl 'i Wov . k 

wiiKou-t the "UseDatabase" 

£ohtex-1 mahagcv. 
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8 a little blt of class 



Abstracting Behavior 

and State 




Classes letyou bundle code behavior and state together. 

In this chapter, you’re setting your webapp aside while you learn about creating Python 
classes. You’re doing this in order to get to the point where you can create a context 
manager with the help of a Python class. As creating and using classes is such a useful 
thing to know about anyway, we’re dedicating this chapter to them. We won’t cover 
everything about classes, but we’ll touch on all the bits you’ll need to understand in order 
to confidently create the context manager your webapp is waiting for. Let’s dive in and see 
what’s involved. 
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context management protocol 


Hooking into the "with" Statement 


At stated at the end of the last chapter, understanding how to hook your setup 
and teardown code into Python’s with statement is straightforward...assuming 
you know how to create a Python class. 

Despite being well over halfway through this book, you Ve managed to get by 
without having to define a class. You Ve written useful and reusable code using 
nothing more than Python’s function machinery. There are other ways to write 
and organize your code, and object orientation is very popular. 

You’re never forced to program exclusively in the object-oriented paradigm when 
using Python, and the language is flexible when it comes to how you go about 
writing your code. But, when it comes to hooking into the with statement, doing 
so through a class is the recommended approach, even though the Standard 
library comes with support for doing something similar without a class (although 
the Standard library 5 s approach is less widely applicable, so we aren’t going to use 
it here). 

So, to hook into the with statement, youil have to create a class. Once you know 
how to write classes, you can then create one that implements and adheres to the 
context management protocol. This protocol is the mechanism (built into 
Python) that hooks into the with statement. 

Let 5 s learn how to create and use classes in Python, before returning to our 
context management protocol discussion in the next chapter. 


Tite context 
management 
protocol lets your 
write a class that 
kooks into the 
"with” statement. 


tliere ,are no 

Dumb Questfens 


Exactly what type of programming language is Python: 
object-oriented, functional, or procedural? 

That’s a great question, which many programmers moving 
to Python eventually ask. The answer is that Python supports 
programming paradigms borrowed from ali three of these popular 
approaches, and Python encourages programmers to mix and 
match as needed. This concept can be hard to get your head around, 
especially if you come from the perspective where ali the code you 
write has to be in a class that you instantiate objects from (as in other 
programming languages like, for instance, Java). 

Our advice is not to let this worry you: create code in whatever 
paradigm you’re comfortable with, but don’t discount the others 
simply because—as approaches—they appear alien to you. 



So...is it wrong to always start by creating a class? 


No, it isn’t, if thafs what your application needs. You don’t have 
to put all your code in classes, but if you want to, Python won’t get in 
your way. 


So far in this book, weVe gotten by without having to create a class, 
but we’re now at the point where it makes sense to use one to solve 
a specific application issue we’re grappling with: how best to share 
our database Processing code within our webapp. We’re mixing and 
matching programming paradigms to solve our current problem, and 
thafs OK. 
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Aw Object-Oriented Priwer 

Before we get going with classes, it’s important to note that we doni 
intend to cover everything there is to know about classes in Python in 
this chapter. Our intention is merely to show you enough to enable you 
to confidently create a class that implements the context management 
protocol. 

Therefore, we won’t discuss some topics that seasoned practitioners of 
object-oriented programming (OOP) might expect to see here, such as 
inheritance and polymorphism (even though Python provides support for 
both). That’s because we’re primarily interested in encapsulation when 
creating a context manager. 

If the jargon in that last paragraph has put you in a blindpanic : doni 
worry: you can safely read on without knowing what any of that OOP- 
speak actually means. 

On the last page, you learned that you need to create a class in order to 
hook into the with statement. Before getting to the specifics of how 
to do that, leti look at what constitutes a class in Python, writing an 
example class as we go. Once you understand how to write a class, weil 
return to the problem of hooking into the with statement (in the next 
chapter). 



Don 5 t be 
freaked out 
by all the 
buzzwords 
on this page! 

If we were to run a competition 
to determine the page in this 
book with this most buzzwords 
on it, this one would win hands- 
down. Doni be put off by all the 
jargon used here, though. If you 
already know OOP, this should 
all make sense. If not, the 
really important bits are 
shown below. Doni worry: all 
this will become clearer as you 
work through the example on the 
next few pages. 


A class buwdles behavior and state 


Using a class lets you bundle behavior and state together in an object. 

When you hear the word behavior , think function —that is, a chunk of code 
that does something (or implements a behavior , if you prefer). 

When you hear the word state , think variables —that is, a place to store 
values within a class. When we assert that a class bundles behavior and 
state together , we’re simply stating that a class packages functions and 
variables. 


A dlass Wndles toythev 

behaviov- a»d state- \ 


The upshot of all of the above is this: if you know what a function is and 
what variables are, you’re most of the way to understanding what a class 


is fas well as how to create one 


pavior 


Classes have methods and attributes 

In Python, you define a class behavior by creating a method. 

The word method is the OOP name given to a function that’s defined 
within a class. Just why methods areni simply known as class functions 
has been lost in the mists of time, as has the fact that class variables areni 
referred to as such—they are known by the name attribute. 


Siciie 


The class 


Methods (a.k.a. 
functions) 


Attributes (a.k.a. 
variables) 
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class makes object 


Creating Objects frow Classes 


To use a class, you create an object from it (you’11 see an example of this below). This is 
known as object instantiation. When you hear the word instantiate , think invoke\ that 
is, you invoke a class to create an object. 

Perhaps surprisingly, you can create a class that has no state or behavior, yet is stili a 
class as far as Python is concerned. In effect, such a class is empty. Let 5 s start our class 
examples with an empty one and take things from there. We’11 work at the interpretebs 
>>> prompt, and you’re encouraged to follow along. 

We begin by creating an empty class called CountFromBy. We do this by prefixing the 
class name with the class keyword, then providing the suite of code that implements 
the class (after the obligatory colon): 

Don’1 formet the Colon. 




Classes stari 
v/rth ita “dass 

dlass suite- 


»> class CountFromBy: 
pass 



The ir>ame of *the dass 


Note how this class’s suite contains the Python keyword pass, which is Python’s empty 
statement (in that it does nothing). You can use pass in any place the interpreter 
expects to find actual code. In this case, we aren’t quite ready to fili in the details of the 
CountFromBy class, so we use pass to avoid any syntax errors that would normally 
resuit when we try to create a class without any code in its suite. 

Now that the class exists, let’s create two objects from it, one called a and another called 
b. Note how creating an object from a class looks very much like calling a function: 


N M • (• I 

pass is a valid 
statement (i.e., it 
is syntacticaUy 
correct), l>ut it 
does nothing. 

Think ol it 
as an empty 
statement. 


»> a = CountFromBy () 
»> b = CountFromBy () 


| ^ These look ,ike tunc-tion calls, doh'-1 -they? 

f Cveaie an objec-t by aypending paven-tbeses -to ibe class name, 
-tben assi^n bbe newly eveated object to a vaviable- 

tliereiare n° _ 

Dumb Questions 


% When l’m looking at someone else’s code, how do I know if something like CountFromBy () is code that creates an object 
or code that calls a function? That looks like a function call to me... 

That’s a great question. On the face of things, you don’t know. However, there’s a well-established convention in the Python 
programming community to name functions using lowercase letters (with underscores for emphasis), while CamelCase (concatenated words, 
capitalized) is used to name classes. Following this convention, it should be ciear that count_f rom_by () is a function call, whereas 
CountFromBy () creates an object. AII is fine just so long as everyone follows this convention, and you’re strongly encouraged to do so, 
too. However, if you ignore this suggestion, all bets are off, and most Python programmers will likely avoid you and your code. 
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Objects Sharc Pehavior but Not State 


When you create objects from a class, each object shares the class’s coded 
behaviors (the methods defined in the class), but maintains its own copy of any 
state (the attributesk 




State 


An object 



Methods (shared with 
all objects created from 
the same class) 


Attributes (*not* shared 
with other objects 
created from the same 
class) 


objett tveated tv-om 
-the same tlass ta» attess 
•the tlasss methods (the 
sKared tode) iWvev, eath 
objett mamtams its 

the attvibutes- 


This distinction will make more sense as we flesh out the CountFromBy example. 


Pef iwiwg what m wawt CoimtFrow&y to do 

Let’s now define what we want the CountFromBy class to actually do (as an empty 
class is rarely useful). 

Let’s make CountFromBy an incrementing counter. By default, the counter will start 
at 0 and be incremented (on request) by 1. We’ll also make it possible to provide an 
alternative starting value and/or amount to increment by This means you’11 be able to 
create, for example, a CountFromBy object that starts at 100 and increments by 10. 

Let’s preview what the CountFromBy class will be able to do (once we have written 
its code). By understanding how the class will be used, you’11 be better equipped to 
understand the CountFromBy code as we write it. Our first example uses the class 
defaults: start at 0, and increment by 1 on request by calling the increase method. 
The newly created object is assigned to a new variable, which weVe called c: 


Note: tltis new 
"C ountFromBy” 

class doesn’t 
exist just yet. 

You’11 create it 

in a little kit. 


The 

startmj-^ 

vdlue is 0- 


»> c = CountFromBy () 
»> c 


Create ahotber new objeet, and assia* it 
to a* object £alled V. 


0 

»> c.increase () 

»> c. increase () • ^ 

»> c. increase () 

»> c 


|«voke the 'mtrease” ">ethod 
U mtvemea the value ot the 
tountev bY one eath time- 



A-Ctev the thv-ee talis to the "mtv-ease” method, 
the value ot the objett is nou/ thv-ee- 
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more with objects 


Poing More with CountFrowPy 

The example usage of CountFromBy at the bottom of the last page 
demonstrated the default behavior: unless specified, the counter maintained by 
a CountFromBy object starts at 0 and is incremented by 1. It’s also possible to 
specify an alternative starting value, as demonstrated in this next example, where 
the count starts from 100: 


The 

startmj-^ 

value is IOO. 


>» 

»> 

100 

»> 

»> 

»> 

»> 

103 


d = CountFromBy(100) 


d 


Me* evea-tihg this h ew 

obieei spcei-Py the stav-ti*a 
value. J 


d. increase () |nvoke the w 'mtvease wethod 

d. increase () > -to the value of 

d. increase () the dountcir by one eath ti^c- 

d 

^_^tev the thv-ee talis to the w intrease"’ r^ethod, 

the value of the w d w objett is now \OZ. 


As well as specifying the starting value, it’s also possible to specify the amount to 
increase by, as shown here, where we start at 100 and increment by 10: 


f * 

V starts 
at IOO, 

and ends 

up at 130- 


»> e = CountFromBy (100, 10) 
»> e 


100 

»> for i in range(3) : 
e.increase() 


i 



»> e 
130 


^-_____ %edities bo-th the startin» 

» -»* -a, 

to intv-emeht by 

Uoke the "mdv-ease" method thvee 
tw es Yiithiw a tt W’ loop, 'mdrevoeintmj 

the value of u e" b 7 10 eadh ti 


In this hnal example, the counter starts at 0 (the default), but increments by 
15. Rather than having to specify (0, 15) as the arguments to the class, this 

example uses a keyword argument that allows us to specify the amount to 
increment by, while leaving the starting value at the default (0): 


starts 

at 0, and 
ends up at 
¥>■ 

v -> 


»> 

»> 

0 

»> 


»> 

45 


f = CountFromBy(increment=l5) 



f 


for j in range 
f.increase() 




As b tfort) 
dall 'Wvtase" 

-thvcc *tiwcs. 


Spc^i-Pics -the a^oun-1 
“to by 


f 
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lf$ Worth Rcpeatiwg Oursdvcs: Objects 
Sharc Pehavior but Not State 


The previous examples created four new CountFromBy objects: c, d, e, and f, each 
of which has access to the increase method, which is a behavior that 5 s shared by 
all objects created from the CountFromBy class. There 5 s only ever one copy of the 
increase method’s code, which all these objects use. However, each object maintains 
its own attribute values. In these examples, that’s the current value of the counter, 
which is different for each of the objects, as shown here: 


»> c 
3 

»> d 
103 
»> e 
130 
»> f 
45 







Th«e towr "Co 
obje^ai^ iM t hei , J n 

attmibuxc values. 


Here’s the key point again : the method code is shared, but the attribute data isn’t. 

It can be useful to think of a class as a “cookie-cutter template” that is used by a 
factory to churn out objects that all behave the same, but have their own data. 


Class kekavior is 
skarect ky eack 
ol its okjects, 
wkereas state is 
not. Eack okject 
maintains its 
own state. 



The ts cbo^\j 
pvirwed with 
'Conh-tFvo rw{ 
(r-eady io 


The Object 
Factory 


IV- OVM 
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methods in action 


Ihvokiwg a Method: Understawd the Petails 

We stated earlier that a method is a function defined within a class. We also saw examples 
of a method from CountFromBy being invoked. The increase method is invoked 
using the familiar dot notation: 


TKe objefrt 


c.increase() 





The hame o-P the method 


Spot the dot- 

It is instructive to consider the code the interpreter actually exeeutes (behind the scenes 
when it encounters the above line. Here is the call the interpreter always turns the 
above line of code into. Note what happens to c: 


The hame o-P the elass 
withih whieh the method 
is de-Pihed 




CountFromBy.increase(c) 

/ 

%ot the dot- 


The °h eC t ( i° mCrease) 


The hame <y P the method 



Yes, it does. But nobody ever does that. 

And neither should you, as the Python interpreter does 
this for you anyway...so why write more code to do 
something that can be written more succinctly? 

Just why the interpreter does this will become clearer as 
you learn more about how methods work. 
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Method Invocation: What Actually Happens 

At first sight, the interpreter turning c . increase () into CountFromBy . increase (c) 
may look a little strange, but understanding that this happens helps explain why every method 
you write takes at least one argument. 

It 5 s OK for methods to take more than one argument, but the first argument always has to 
exist in order to take the object as an argument (which, in the example from the last page, is 
c). In fact, it is a well-established practice in the Python programming community to give each 
method’s first argument a special name: self. 

When increase is invoked as c.increase (), you’d imagine the method’s def line should 
look like this: 

def increase(): 


However, defining a method without the mandatory first argument will cause the interpreter to 
raise an error when your code runs. Consequently, the increase method’s def line actually 
needs to be written as follows: 


def increase(self): 


It is regarded as very bad form to use something other than the name self in your 
class code, even though the use of self does take a bit of getting used to. (Many other 
programming languages have a similar notion, although they favor the name this. Python’s 
self is basically the same idea as this.) 

When you invoke a method on an object, Python arranges for the first argument to be the 
invoking object instance, which is always assigned to each methodh self argument. This fact 
alone explains why self is so important and also why self needs to be the first argument to 
every object method you write. When you invoke a method, you don’t need to supply a value for 
self, as the interpreter does this for you: 


When writingf 
code in a 
class, tkink 
ol "sell” as an 
alias to the 
current object. 


What you write: 


What Python executes: 


The dass 


d.increase() 



There* s y\o need 
bo supply a 
value -ror W sel-C* • 


Now that you Ve been introduced to the importance of self, let 5 s take a look at writing the 
code for the increase method. 



The value o£ 
w d** is assi^ed 
bo M sel'f w by the 
mtev-pvetev. 
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Adding a Method to a Class 

Lefs create a new file to save our class code into. Create countf romby. py, then 
add in the class code from earlier in this chapter: 

class CountFromBy: 
pass 


We’re going to add the increase method to this class, and to do so we’ll remove the 
pass statement and replace it with increase’s method definition. Before doing this, 
recall how increase is invoked: 


c.increase() 


Based on this call, you’d be forgiven for assuming the increase method takes no 
arguments, as there 5 s nothing between the parentheses, right? However, this is only 
half true. As you just learned, the interpreter transforms the above line of code into 
the following call: 

CountFromBy.increase(c) 


The method code we write needs to take this transformation into consideration. With 
all of the above in mind, here’s the def line for the increase method that we’d use 
in this class: 


Me-tKods are 
Wce so 

ave detmed "d* 

"de-C”. 


class CountFromBy: 

def increase(self) -> None: 



As witb tV>e °ther WWs 

ihis book, we provide an 
,-ta-tion tov tbe ve-twrn valve. 


f ^ ygvmenl -to evevy method is always "sel-f" 
and rts value » supplied by the intevpvetev. ^ 


There are no other arguments to the increase method, so we do not need to 
provide anything other than self on the def line. However, it is vitally important 
that we include self here, as forgetting to results in syntax errors. 


With the def line written, all we need to do now is add some code to increase. 
Lefs assume that the class maintains two attributes: val, which contains the current 
value of the current object, and incr, which contains the amount to increment val 
by every time increase is invoked. Knowing this, you might be tempted to add this 
incorrect line of code to increase in an attempt to perform the increment: 


val += incr 

But here’s the correct line of code to add to the increase method: 


class CountFromBy: 

def increase(self) -> None: 
self.val += self.incr 



Why do you think this line of code is correct, whereas the previous was incorrect? 


£u\nrehi value o( u val” 
and mdvease i-t by 
-the value of Wr" 
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a little bit of class 


Are You Serioos About "self"? 


o 



Wait a minute...1 thought one of Python's big 
wins was that it s code is easy to read. I f ind that 
use of "self" anything but easy on the eye, and the 
fact that ifs part of classes (which must get a lot 
of use) has me thinking: seriously?!? 


Don’t worry. Getting used to self won’t take long. 

We agree that Python’s use of self does look a little weird...at 
first. However, over time, you’11 get used to it, so much so that you’11 
hardly even notice it 5 s there. 

If you completely forget about it and fail to add it to your methods, 
you’11 know pretty quickly that something is amiss—the interpreter 
will display a slew of TypeErrors informingyou that something is 
missing, and that something is self. 

As to whether or not the use of self makes Python’s class code 
harder to read...well, we’re not so sure. In our mind, every time 
we see self used as the first argument to a function, our brains 
automatically know that we’re looking at a method, not a function. 
This, for us, is a good thing. 

Think of it this way: the use of self indicates that the code you’re 
reading is a method, as opposed to a function (when self is not 
used). 


you are here ► 
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self == object 


The Iwportawce of "self" 


The increase method, shown below, prefixes each of the class 5 s attributes with 
self within its suite. You were asked to consider why this might be: 


class CountFromBy: 

def increase(self) -> None: 
self.val += self.incr 


Wats -the deal wi£h 
Vlt" v/i-thin 
the methods suite? 


You already know that self is assigned the current object by the interpreter 
when a method is invoked, and that the interpreter expects each methodi first 
argument to take this into account (so that the assignment can occur). 

Now, consider what we already know about each object created from a class: it 
shares the class’s method code (a.k.a. behavior) with every other object created 
from the same class, but maintains its own copy of any attribute data (a.k.a. state). 
It does this by associating the attribute values with the object—that is, with self. 


Knowing this, consider this version of the increase method, which, as we said 
a couple of pages ago, is incorrect: 

class CountFromBy: porrt do ^ 

def increase (self) -> None: do 7 olA » 

val += incr ^_ should- 



On the face of things, that last line of code seems innocent enough, as all it does 
is increment the current value of val by the current value of incr. But consider 
what happens when this increase method terminates: val and incr, which 
exist within increase, both go out of scope and consequently are destroyed the 
moment the method ends. 



Whoops. That’s our bad... 

We slipped in that statement about scope without 
much explanation, didn’t we? 

In order to understand what has to happen when 
you refer to attributes in a method, let’s first spend 
some time understanding what happens to variables 
used in a function. 
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a little bit of class 


Copiwg with Scopmg 

To demonstrate what happens to variables used within a function, let’s experiment 
at the >>> prompt. Try out the code below as you read it. WeVe numbered the 
annotations 1 through 8 to guide you as you follow along: 


souhdbi-fce ; 
TUhdiioh aUcfls 

a single 

3. TV»e avgwien-t 

is assig»>ed to 
3*i<rthev vatiaWe 
inside the WW- 



~~l- Atfcev — 
the -Pund-tio» 
displays -the 
souhdbi-fce, 
value o-p 

is stili 
3££essible. 


0. B«*t nome 

■the vaviahles wsed 
yjithm the twitW 
ave attessihle, 
as they only 
evtist Within the 
•funfctions suite- 




\ 


K 


Python 3.5.1 Shell 

»> 

»&' def soundbite (from_outside) : 
insider = 'James 
outsider = from_outside 
print (from_outside, insider, outsider) 



5 ' ^ v ^ e 'S assigned to 
a vairiab, e Mlled ‘W" 


»> name = 'Bond' ^ 

»> soundbite (name) 

Bond James Bond ^ w souy^db'»‘te ,, 

»> name WW *«s mvoked- 

1 Bond' 

»> insider 

Traceback (most recent call last): 

File "<pyshell#29>", line 1, in <module> 
insider 

NameError: name 'insider' is not defined 
»> outsider 

Traceback (most recent call last): 

File "<pyshell#30>", line 1, in <module> 
outsider 

NameError: name 'outsider' is not defined 
»> from_outside 

Traceback (most recent call last): 

File "<pyshell#31>", line 1, in <module> 
from_outside 

NameError: name 'from_outside' is not defined 

»> 

»> 


Ln: 83 Coi: 4 



2- $ value is 
^sighcd {x> 

& vaHablc 
inside -the 
-Puh^-tioh. 


The ^ 

-funfction s 
vav-iables 
ave used “to 
display a 

messa^e. 


When variables are defined within a function’s suite, they exist while the function 
runs. That is, the variables are “in scope,” both visible and usable within the function’s 
suite. However, once the function ends, any variables defined within the function are 
destroyed—they are “out of scope,” and any resources they used are reclaimed by the 
interpreter. 


This is what happens to the three variables used within the soundbite function, as 
shown above. The moment the function terminates, insider, outsider, and 
f rom_outside cease to exist. Any attempt to refer to them outside the suite of 
function (a.k.a. outside the function’s scope) results in a NameError. 


you are here ► 
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back to self 


Pref ix Your Attribute Names with "self" 

This function behavior described on the last page is fine when you’re dealing with a 
function that gets invoked, does some work, and then returns a value. You typically 
don’t care what happens to any variables used within a function, as you’re usually only 
interested in the function’s return value. 

Now that you know what happens to variables when a function ends, it should be ciear 
that this (incorrect) code is likely to cause problems when you attempt to use variables 
to store and remember attribute values with a class. As methods are functions by 
another name, neither val nor i ner will survive an invocation of the increase 
method if this is how you code increase: 


class CountFromBy: pov/t do “this, as “tbese 

def increase(self) -> None: variables aorv^e 

val += incr _ or>te -^Ke metbod ends- 



However, with methods, things are different. The method uses attribute values that 
belong to an object, and the object 5 s attributes continue to exist after the method 
terminates. That is, an object’s attribute values are not destroyed when the 
method terminates. 


In order for an attribute assignment to survive method termination, the attribute 
value has to be assigned to something that doesn’t get destroyed as soon as the 
method ends. That something is the current object invoking the method, which is 
stored in self, which explains why each attribute value needs to be prefixed with 
self in your method code, as shown here: 

class CountFromBy: 

def increase(self) -> None: 
self.val += self.incr 



n ' S f 'T* be ^, as "v 
mdr ar e * ow a ss oCla 

w-th ih e oWti thanks io 
the use ot «sel-P" 


The rule is straightforward: if you need to refer to an attribute in your class, you 
must prefix the attribute name with self. The value in self as an alias that 
points back to the object invoking the method. 

In this context, when you see self, think “this objecfs.” So, self . val can be 
read as “this objecfs val.” 


« 


se 


It" is 


dh 



alias -to -the 

object. 


An object 


Methods (shared with 
all objects created from 
the same class) 


Attributes (*not* shared 
with other objects 
created from the same 
class) 
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a little bit of class 


Initialize (Attribute) Values Pefore Use 


All of the discussion of the importance of self sidestepped an important issue: how are 
attributes assigned a starting value? As it stands, the code in the increase method—the 
correct code, which uses self—fails if you execute it. This failure occurs because in 
Python you can’t use a variable before it has been assigned a value, no matter where the 
variable is used. 


To demonstrate the seriousness of this issue, consider this short session at the >>> prompt. 
Note how the first statement fails to execute when either of the variables is undefined: 


|Jp you -bry ■fco e%efcu*be 
Codt bhai v-e-Pev-s -to - 
unmrbaliz-cd vav-iables... 


•••"the ih‘tev"pv~c‘tcv' —^ 
^omplaihs. 


v 

»> val += incr 

Traceback (most recent call last): 

File "<pyshell#l>", line 1 r in <module> 
val += incr 

NameError: name 'val’ is not defined <:— 


Assiy» a value b> -^ >» val _ 0 

"val", ther. tv-y a$a'm- 


• ahd -the 

nvfcev-pvetev -^ 

^omplai^s agaih/ 


»> val += incr 

Traceback (most recent call last): 

File M <pyshell#3>”, line 1, in <module> 
val += incr 

NameError: name 'incr' is not defined <— 


Assi <>jr\ a ue *to »> incr — 1 

\*0c\ a*d -bry a^am- 


A s val is Uhdc-fihcd/ 

the iivteirpvc-tev- 
vc-Puses -to nm the 
1'^e o-P £ode. 


As W is 
unde-Pined, the 
'mtevpv-etev demtmues 
to v-e-Puse to vuk the 
Ime o-P £ode. 


•ahd it woirked 
this time. 


»> val += incr 



V 


h bo-th val and w have values U, they 
av-e mrtiali«dt the mte^ e te, is ha PP y t l t 
theur values w,thout raismg a Na»cSrJl 


No matter where you use variables in Python, you have to initialize them with a starting 
value. The question is: how do we do this for a new object createdfrom a Python class? 

If you know OOP, the word “constructor” may be popping into your brain right about now. 
In other languages, a constructor is a special method that lets you define what happens 
when an object is first created, and it usually involves both object instantiation and attribute 
initialization. In Python, object instantiation is handled automatically by the interpreter, so 

you don’t need to define a constructor to do this. A magic method called_ init _lets 

you initialize attributes as needed. Let’s take a look at what dunder init can do. 


you are here ► 
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init _ is magic 


Puwdcr "init" Iwitializes Attributes 

Gast your mind back to the last chapter, when you used the dir built-in function 
to display all the details of Flask’s req object. Remember this output? 



Look a*t 
all -tkosc 
dundev-sj 


E 


['_class_'_delattr_'_dict_'_dir_'_doc_'_enter_'_eq_'_exit_'_format_'_ge_ 

'_getattribute_'_gt_'_hash_'_init_'_le_'_lt_'_module_'_ne_'_new_'_reduce_ 

'_reduce_ex_'_repr_'_setattr_'_sizeof_'_str_'_subclasshook_'_weakref_'_get_file_stream', 

'_get_stream_for_parsing', '_is_old_module', '_load_form_data', '_parse_content_type', '_parsed_content_type', 
'accept_charsets', 'accept_encodings', 'accept_languages', 'accept_mimetypes', 'access_route', 'application', 'args', 
'authorization', 'base_url', 'blueprint', 'cache_control', 'charset', 'close', 'content_encoding', 'content_length', 'content_md5', 
'content_type', 'cookies', 'data', 'date', 'dict_storage_class', 'disable_data_descriptor', 'encoding_errors', 'endpoint', 'environ', 
'files', 'form', 'form_data_parser_class', 'from_values', 'full_path', 'get_data', 'get_json', 'headers', 'host', 'host_url', 'if_match', 
'if_modified_since', 'if_none_match', 'if_range', 'if_unmodified_since', 'input_stream', 'is_multiprocess', 'is_multithread', 
'is_run_once', 'is_secure', 'is_xhr', 'json', 'list_storage_class', 'make_form_data_parser', 'max_content_length', 
'max_form_memory_size', 'max_forwards', 'method', 'mimetype', 'mimetype_params', 'module', 'on_json_loading_failed', 
'parameter_storage_class', 'path', 'pragma', 'query_string', 'range', 'referrer', 'remote_addr', 'remote_user', 'routing_exception', 


J l r 


y ? K 


J U 


II_f. 


».! K 


J K 


At the time, we suggested you ignore all those dunders. However, it’s now time 
to reveal their purpose: the dunders provide hooks into every class’s Standard 
behavior. 


Unless you override it, this Standard behavior is implemented in a class called 
object. The object class is built into the interpreter, and every other Python 
class automatically inherits from it (including yours). This is OOP-speak for stating 
that the dunder methods provided by ob j ect are available to your class to use as 
is, or to override as needed (by providing your own implementation of them). 


You don’t have to override any ob j ect methods if you don’t want to. But if, for 
example, you want to specify what happens when objects created from your class 
are used with the equality operator (==), then you can write your own code for 

the_eq_method. If you want to specify what happens when objects are used 

with the greater-than operator (>), you can override the_ge_method. And 

when you want to initialize the attributes associated with your object, you can use 
the_init_method. 

As the dunders provided by ob j ect are so useful, they’re held in near-mystical 
reverence by Python programmers. So much so, in fact, that many Python 
programmers refer to these dunders as the magic methods (as they give the 
appearance of doing what they do “as if by magic”). 

All of this means that if you provide a method in your class with a de f line like 
the one below, the interpreter will call your init method every time you 
create a new object from your class. Note the inclusion of self as this dunder 
init’s flrst argument (as per the rule for all methods in all classes): 


def 


init (self) 


The Standard 
dunder 
methods, 
availahle to 
all classes, are 
known as "the 
magic methods.” 

Despi-te s-fo-ahge-lookmg name, 
duhdev "mi-fc" is a method like a»y 

«, r T ^ erne ^ : y<* ">«si pass 
sel+ as its -fivst av-gument. 
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a little bit of class 


Iwitializmg Attributes with Puwder "init" 

Let 5 s add_ init _to our CountFromBy class in order to initialize the objects 

we create from our class. 

For now, let 5 s add an empty _ init _method that does nothing but pass (we’11 

add behavior in just a moment): 

"' 0, *' c,r rfc; “this duhder 
'mi-t doesnt do anything. 
Howevev, the use J " se lt" as its 
+iirst argument is a Bl<$ CLUB 
that dunder “init” is a method- 


class CountFromBy: 

def _init_(self) -> None: 

pass 

def increase(self) -> None: 
self.val += self.incr 


We know from the code already in increase that we can access attributes in our 
class by prefixing their names with self. This means we can use self . val and 

self.incrto refer to our attributes within_ init _, too. However, we want 

to use_ init _to initialize our classi attributes (val and incr). The question 

is: where do these initialization values come from and how do their values get into 
init ? 


Pass any awount of argument data to dunder "init" 

As_init _is a method, and methods are functions in disguise, you can pass as 

many argument values as you like to_ init _(or any method, for that matter). 

All you have to do is give your arguments names. Let 5 s give the argument that well 
use to initialize self . val the name v, and use the name i for self.incr. 

Let 5 s add v and i to the def line of our_ init _method, then use the values 

in dunder init 5 s suite to initialize our class attributes, as follows: 


class CountFromBy: 

def _init_(self, v: int, i: 

self.val = v 
self.incr = i 

def increase(self) -> None: 

Use the values ot V and self. val += self. incr 

V -to initiali** the tla» s 
a-bbributes (vAiidh are selt-val 

a*d "seltmtr", resfefctivelyl 




Add V a»d V as 

argument* to dunder 


If we can now somehow arrange for v and i to acquire values, the latest version 

of_ init _will initialize our class’s attributes. Which raises yet another 

question: how do we get values into v and i? To help answer this question, we 
need to try out this version of our class and see what happens. Let’s do that now. 


you are here ► 
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try your class 



Using the edit window in IDLE, take a moment to update the code in your countf romby. py file 
to look like that shown below. When you’re done, press F5 to start creating objects at IDLE’s »> 
prompt: 


Pvess F5 -to 
OU-t 

dlass m IPL-E s 
sVkdl* 


-> 


countfromby.py - /Users/paul/Desktop/NewBook/ch08/countfromby.py (3.5.1) 


class CountFromBy : 


def 


init (self, v: int, i: int) -> None 


self.val = v 
self.incr = i 


def increase (self) -> None: 
self.val += self.incr 



Ln: 2 Coi: 0 


The Idies-fc 
versi ok o-P ouv* 

'CouhiFlrornBy” 

dlass. 


Creaie a hew 
objefrt Ualled 

"g'0 f<rom -the 

dlass...bwt when 

you do -fchis, you 
jct ah error/ 



Pressing F5 exeeutes the code in the edit window, which imports the CountFromBy class into the 
interpreter. Look at what happens when we try to create a new object from our CountFromBy class: 


# # # Python 3.5.1 Shell 

Python 3.5.1 (v3.5.1:37a07cee5969, Dec 5 2015, 21:12:44) 

[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin 

Type "Copyright", "credits" or "license()" for more information. 

»> 

========= RESTART : /Users/paul/Desktop/_NewBook/ch07/countfromby.py ====== 

»> 

»> g = CountFromBy () 

Traceback (most recent call last): 

File "<pyshell#l>", line 1, in <module> 
g = CountFromBy() 

TypeError: _init_() missing 2 required positional arguments: 'v' and 'i' 

»> 

»> 


Ln: 13 Coi: 4 



This may not have been what you were expecting to see. But take a look at the error message (which 
is classed as a TypeError), paying particular attention to the message on the TypeError line. 

The interpreter is telling us that the_ init _method expected to receive two argument values, v 

and i, but received something else (in this case, nothing). We provided no arguments to the class, 
but this error message telis us that any arguments provided to the class (when creating a new object) 
are passed to the_ init _method. 

Bearing this in mind, let’s have another go at creating a CountFromBy object. 
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a little bit of class 


Let’s return to the »> prompt, and create another object (called h) that takes two integer values as 
arguments for v and i: 

"TypeEvW' -this i, 




Python 3.5.1 Shell 


»> 

»> h = CountFromBy (100, 10) 
»> h.val 
100 

Vou tam »> h. incr 

the s* 10 



attcss 
value “the 

V objetVs 


»> h. increase () 
»> h.val 
110 



Imvokmg the u im£\rease” method 
does what you expett it to do. Ii 
inc<rer»c*k "h.val" by the a«.ouht i 
“hW\ ' 


>» h 

<_main_.CountFromBy object at 0xl05al3da0> 

»> 

»> 



Vov v/ev-e pvobably expett'm<) to see tt HO' dts^layed heve, 
I L • ..i 4-w.< fvath^ir erwtie) n>essa<\e mstead- 


Ln: 37 Coi: 4 


As you can see above, things work better this time, as the TypeError exception is gone, which 
means the h object was created successfully. You can access the values of h’s attributes using 
h. val and h. incr, as well as call the objecfs increase method. Only when you try to access 
the value of h do things get strange again. 

What have we Icarwed f rom this Test Prive? 


Here are the main takeaways from this Test Drive : 

■ When you’re creating objects, any argument values provided to the class are passed to the_ init _method, as 

was the case with 100 and 10 above. (Note that v and i cease to exist as soon as dunder init ends, but we aren’t 
worried, as their values are safely stored in the objecfs self. val and self. incr attributes, respectively.) 

■ We can access the attribute values by combining the objecfs name with the attribute name. Note how we used 

h. val and h. incr to do this. (For those readers coming to Python from a “stricter” OOP language, note that we 
did this without having to create getters or setters.) 

■ When we use the object name on its own (as in the last interaction with the Shell above), the interpreter spits back a 
cryptic message. Just what this is (and why this happens) will be discussed next. 


you are here ► 
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control your repr 


Uwdcrstawdmg CouwtFrowPy'$ Represewtatiow 

When we typed the name of the object into the shell in an attempt to display its current 
value, the interpreter produced this output: 

tW-fc woirvy i-f y ou 

have a di-P-fereh-t 
value hev-e. AII will 
beeonne tlear befov* 
tbe emd ot -this 

P a 9 e ' 

In the session below, note how the strange message displayed for j is made up of values 
that are produced when we call certain built-in functions (BIFs). Follow along with the 
session first, then read on for an explanation of what these BIFs do: 


<_main_.CountFromBy object at 0xl05al3da0> 

We described the above output as “strange,” and on first glance, it would certainly 
appear to be. To understand what this output means, let’s return to IDLE 5 s shell and 
create yet another object from CountFromBy, which due to our deeply ingrained 
unwillingness to rock the boat, we’re calling j. 


Python 3.5.1 Shell 


»> 

»> j = CountFromBy (100 , 10) 

»> j 

< main 


.CountFromBy object at 0xl035be278> 
\ 


»> 

»> type(j) 

<class 1 main_.CountFromBy'> 

»> 

»> id(j) 

4351320696 

»> 

»> hex (id (j)) 

1 0xl035be278’ 

»> 

»> 



ou-tpu-t (or 
J ' s up o-P 

values pvodueed 
by some o-f 
PytU' s BIFs. 


Ln: 21 Coi: 4 


The type BIF displays information on the class the object was created from, reporting 
(above) that j is a CountFromBy object. 

The id BIF displays information on an objects memory address (which is a unique 
identifier used by the interpreter to keep track of your objects). What you see on your 
screen is likely different from what is reported above. 

The memory address displayed as part of j 5 s output is the value of id converted 
to a hexadecimal number (which is what the hex BIF does). So, the entire message 
displayed for j is a combination of type 5 s output, as well as id’s (converted to 
hexadecimal). 

A reasonable question is: why does this happen? 

In the absence of you telling the interpreter how you want to represent your objects, 
the interpreter has to do something , so it does what 5 s shown above. Thankfully, you can 
override this default behavior by coding your own_repr_magic method. 
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dunder repr 
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your ohjects are 
represented ky 
the interpreter. 













a little bit of class 


Pef miwg CouwtFrowPy'$ Representation 


As well as being a magic method, the_ repr _functionality is also available as 

a built-in function called repr. Here’s part of what the help BIF displays when 
you ask it to teli you what repr does: “Return the canonical string representation 
of the object.” In other words, the help BIF is telling you that repr (and by 
extension,_ repr _) needs to return a stringified version of an object. 

What this “stringified version of an object” looks like depends on what each 
individual object does. You can control what happens for your objects by writing 

a_ repr _method for your class. Lefs do this now for the CountFromBy 

class. 


Begin by adding a new def line to the CountFromBy class for dunder repr, 
which takes no arguments other than the required self (remember: ifs a 
method). As is our practice, lefs also add an annotation that lets readers of our 
code know this method returns a string: 


Like evevy o-thev we-thod def 

Wll wnte, tW.s eme U 

-take m*to “t-bat c 

aUays yvovides 
a value -fov ‘the Fivs-b 


3Y-gur*c*i 



repr (self) 


-> str: 



This lets users o( this 

*»ethod khow that this 
Fuhetioh ihtehds to vetu^h 
5 sthhg. Remembeir: using 

3hh °t^tiohs ih your toAt 
,s optiohal, but help-Pul. 


With the de f line written, all that remains is to write the code that returns a string 
representation of a CountFromBy object. For our purposes, all we want to do 
here is take the value in self . val, which is an integer, and convert it to a string. 

Thanks to the str BIF, doing so is straightforward: 


def _repr_(self) -> str: 

return str(self.val) 



When you add this short function to your class, the interpreter uses it whenever 
it needs to display a CountFromBy object at the >>> prompt. The print BIF 
also uses dunder repr to display objects. 

Before making this change and taking the updated code for a spin, lef s return 
briefly to another issue that surfaced during the last Test Drive. 


Take -the value i* 
“sel-f-vaf*, ‘b*™ 
mto a sbrmj» a*d 

-the* v-e-tuv* i*t 
-to "this method ^ 
eallev. 
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countfromby does more 


Providiwg Scwsiblc Pefaults for CouwtFrowPy 

Let’s remind ourselves of the current version of the CountFromBy class 5 s_ init 

method: 


def _init_(self, v: int, i: int) -> None: 

self.val = v 
self.incr = i 


Recall that when we tried to create a new object from this class without passing values 
for v and i, we got a TypeError: 


TWis vevsio* o-f the 
duvdev "mit” ™ethod 
en^efrts tw o avgumewt 
values to be pvovided 
evevy time it is mvoked- 



Earlier in this chapter, we specified that we wanted the CountFromBy class to 
support the following default behavior: the counter will start at 0 and be incremented 
(on request) by 1. You already know how to provide default values to function 
arguments, and the same goes for methods, too—assign the default values on the de f 
line: 

• • • 

def _init_(self, v: int=0, i: int=l) -> None: 

self.val = v 
self.incr = i 



If you make this small (but important) change to your CountFromBy code, then save 
the file (before pressing F5 once more), youil see that objects can now be created with 
this default behavior: 


As methods ave Wtions, 

p y su PP<*"fc the use 
°t de-Pault values (or 
avguments (although weVe 
sdovihg a B- heve tov ouv 
use o+ single-chavaetev 
vaviable Kames- "v" \ s -t^e 
value, wheveas "i" is the 
mdv-emehtmg value). 


Python 3.5.1 Shell 


»> 

»> 

»> 

»> 

0 

»> 

1 

»> 

»> 

1 

»> 

»> 


i = CountFromBy() 
i. val 

i.incr 

i.increase() 
i .val 


This ali wovks as e^edted, with the ‘Wvease 
method mdvemevted "i val' by o*e eath W .ts 
mvoked- This is the detault behav.ov. 


Ln: 50 Coi: 4 
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a little bit of class 



Make sure your class code (in countfromby .py) is the same as ours below. With your class code 
loaded into IDLE’s edit window, press F5 to take your latest version of the CountFromBy class for 
a spin: 


This is the 
"CouftlFvomB7" tlass 

>witb the tode V 
dunder 'Vep* added 



# # # countfromby.py -/Users/paul/Desktop/NewBook/ch08/countfromby.py (3.5.1) 


class CountFromBy: 

def _init_ (self, v: int=0, i: int=l) -> None: 

self.val = v 
self.incr = i 

def increase (self) -> None: 
self.val += self.incr 

def _repr_ (self) -> str: 

return str (self.val) 


Ln: 1 3 Coi: 0 



The V objeet 
uses the elass s 
de-fault values, 
whith stavt 
at 0 and ave 
indveased by I- 



The V 


objcet 

pvovides 

altevnative 

values -fov both 
detaults. 


»> 

»> 

0 

»> 

»> 

1 

»> 

1 

»> 

»> 

100 

»> 

»> 

101 

»> 

»> 

100 

»> 

»> 

110 

»> 

»> 

0 

»> 

»> 

15 

»> 


k = CountFromBy() 
k 



jVheh you reter h> ihe obj eti ai 
,, e ^ or in a call io 

' ihe d ^r Vepv" tode r UM . 


print (k) 


1 = CountFromBy(100) 
1 

1.increase() 
print (1) 


m = CountFromBy(100, 10) 
m 

m.increase() 
m 

n = CountFromBy(i=15) 
n 


n.increase() 
n 


The T objett pvovides 
an altevnative stavtin$ 
value, then inevements 
by I eaeh tw 
u intvease is ealled* 


The n objeet uses a 
keywovd avgument to 
pvovide an altevnative 
value to in£ver*ent by 
(but stavts at O). 


Ln: 33 Coi: 4 
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classes so far 


Classes: What We Kwow 


With the CountFromBy class behaving as specified earlier in this chapter, 
let 5 s review what we now know about classes in Python: 



BULLET POINTS 
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Python classes let you share behavior (a.k.a. 
methods) and state (a.k.a. attributes). 

If you remember that methods are functions, and 
attributes are variables, you won’t go far wrong. 

The class keyword introduces a new class in your 
code. 

Creating a new object from a class looks very like a 
function call. Remember: to create an object called 
mycount from a class called CountFromBy, 
you’d use this line of code: 

mycount = CountFromBy() 

When an object is created from a class, the object 
shares the class’s code with every other object 
created from the class. However, each object 
maintains its own copy of the attributes. 

You add behaviors to a class by creating methods. A 
method is a function defined within a class. 

To add an attribute to a class, create a variable. 


■ Every method is passed an alias to the current object 
as its first argument. Python convention insists that 
this first argument is called self. 

■ Within a method’s suite, referrals to attributes are 
prefixed with self, ensuring the attribute’s value 
survives after the method code ends. 

■ The_ init _method is one of the many magic 

methods provided with all Python classes. 

■ Attribute values are initialized by the_ init_ 

method (a.k.a. dunder init). This method lets you 
assign starting values to your attributes when a new 
object is created. Dunder init receives a copy 

of any values passed to the class when an object is 
created. Forexample, the values 100 and 10 are 
passed into_ init _when this object is created: 

mycount2 = CountFromBy(100, 10) 

■ Another magic method is_ repr_ , which allows 

you to control how an object appears when displayed 
at the »> prompt, as well as when used with the 

print BIF. 


Chapter 8 


O 




We wanted to create a context manager. 

We know it’s been a while, but the reason we started down this path 
was to learn enough about classes to enable us to create code that 
hooks into Python’s context management protocol. If we can 

hook into the protocol, we can use our webapp’s database code with 
Python’s with statement, as doing so should make it easier to share 
the database code, as well as reuse it. Now that you know a bit about 
classes, you’re ready to get hooked into the context management 
protocol (in the next chapter). 
















a little bit of class 


Chapter 8$ Code 


TWis 'is -the 
dodc m “the 

py * -file- 



class CountFromBy: 

def _init_(self, v: int=0, i: int=l) 

self.val = v 
self.incr = i 


def increase(self) -> None: 
self.val += self.incr 


def _repr_(self) -> str: 

return str (self.val) 


-> None: 


you are here ► 
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9 the context rn an age m ent protocol 


Hooking into Python’s 

with Statement 




Y es, indeecL.for a small 
fee, we can certainly 
manage the context within 
which your code runs. 


It’s time to take what you’ve just learned and put it to work. 

Chapter 7 discussed using a relational database with Python, while Chapter 8 provided 
an introduction to using classes in your Python code. In this chapter, both of these 
techniques are combined to produce a context manager that lets us extend the with 
statement to work with relational database Systems. In this chapter, you’11 hook into 
the with statement by creating a new class, which conforms to Python’s context 
management protocol. 


this is a new chapter 
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which is best? 


What'$ the Pcst Way to Share Our Webapp'$ 
Patabase Code? 


During Ghapter 7 you created database code in your log_request function that worked, 
but you had to pause to consider how best to share it. Recall the suggestions from the end of 
Ghapter 7: 



0 



O 


Isn't it ciear ifs time we 
considered using classes 
and objects as the correct 
way to handle this type of 
reuse? 


O 

o 



At the time, we proposed that each of these suggestions was valid, but believed Python 
programmers would be unlikely to embrace any of these proposed Solutions on their own. We 
decided that a better strategy was to hook into the context management protocol using the 
with statement, but in order to do that, you needed to learn a bit about classes. They were 
the subject of the last chapter. Now that you know how to create a class, ifs time to retura to 
the task at hand: creating a context manager to share your webapp’s database code. 


336 Chapter 9 






the context management protocol 


Cowsider What YouYe Tryiwg to Po, Revisited 

Below is our database management code from Chapter 7. This code is currently part 
of our Flask webapp. Recall how this code connected to our MySQL database, saved 
the details of the web request to the log table, committed any unsaved data, and then 
disconnected from the database: 


import mysql.connector 

def log_request(req: 'flask_request', res: str) -> None: 
"""Log details of the web request and the resuits.""" 


dbconfig = { 


This bi*t 
uses *the 
dvederrtiaU 
■fco donne^t “to 
•the database, 
then fcv-eates 
a euv-sov. 


'host': '127.0.0.1', 

'user': 'vsearch', 

'password': 'vsearchpasswd ', 

'database': ’vsearchlogDB’, } 


conn = mysql.connector.connect(**dbconfig) 
cursor = conn.cursor() 


This dietionary 
details the 
database ^orme^tioh 
^havaeteHsties. 


SQL = """insert into log 

(phrase, letters, ip, browser_string, resuits) 
values 

( O, _ Q. _ Q. _ Q__ Q, —, \ II II II 

■oS, fb, ^b , '85 ^ '6S/ 


cursor.exeeute( SQL, 


conn.commit() 
cursor.close() 
conn.close() 




(req.form[’phrase'], 
req.form['letters'], 
req.remote_addr , 
req.user_agent.browser, 
res, )) 

this eode 

teairs down the 
database ^onnedtion. 



This is the dode 
that does the 
aetual wovk- 
't adds the 
^c^uest data 

to the lt lo 9 ” 

database table. 


How best to create a context wanager? 

Before getting to the point where you can transform the above code into something 
that can be used as part of a with statement, let’s discuss how this is achieved by 
conforming to the context management protocol. Although there is support for 
creating simple context managers in the Standard library (using the contextiib 
module), creating a class that conforms to the protocol is regarded as the correct 
approach when you’re using with to control some external object, such as a 
database connection (as is the case here). 

With that in mind, let 5 s take a look at what’s meant by “conforming to the context 
management protocol. 

you are here ► 337 





en ter exit init 


Managing Context with Methods 


The context management protocol sounds intimidating and scary, but it 5 s actually 
quite simple. It dictates that any class you create must define at least two magic 

methods:_enter_and_exit_. This is the protocol. When you adhere to the 

protocol, your class can hook into the with statement. 

Puw der "enter" perforws setup 

When an object is used with a with statement, the interpreter invokes the object’s 

_enter_method before the with statement 5 s suite starts. This provides an 

opportunity for you to perform any required setup code within dunder enter. 

The protocol further States that dunder enter can (but doesn’t have to) return a value 
to the with statement (youil see why this is important in a little bit). 


A protocol is an 
agreed procedure 
(or set ol rules) 
that is to te 
adhered to. 


Puwder "exit" does teardown 


As soon as the with statemenhs suite ends, the interpreter always invokes the object’s 

_exit_method. This occurs after the with 5 s suite terminates, and it provides an 

opportunity for you to perform any required teardown. 

As the code in the with statement’s suite may fail (and raise an exception), dunder 
exit has to be ready to handle this if it happens. We’ll return to this issue when we 
create the code for our dunder exit method later in this chapter. 

If you create a class that defines_enter_and_exit_, the class is 

automatically regarded as a context manager by the interpreter and can, as a 
consequence, hook into (and be used with) with. In other words, such a class conforms 
to the context management protocol, and implements a context manager. 

(As you know) dunder "init" initializes 

In addition to dunder enter and dunder exit, you can add other methods to your 

class as needed, including defining your own_init_method. As you know from 

the last chapter, defining dunder init lets you perform additional object initialization. 
Dunder init runs before _enter_(that is, before your context manager’s setup code executes ). 

It’s not an absolute requirement to define_init_for your context manager (as 

_enter_and_exit_are all you really need), but it can sometimes be 

useful to do so, as it lets you separate any initialization activity from any setup activity. 
When we create a context manager for use with our database connections (later in 

this chapter), we define_init_to initialize our database connection credentials. 

Doing so isn’t absolutely necessary, but we think it helps to keep things nice and tidy, 
and makes our context manager class code easier to read and understand. 


II your class 
delines dunder 
" enter” and 
dunder "exit”, 
it’s a context 
manager. 
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the context management protocol 


YouVe Already Seew a Context Manager in 
Aetion 

You first encountered a with statement back in Chapter 6 when you used one to 
ensure a previously opened file was automatically closed once its associated with 
statement terminated. Recall how this code opened the todos.txt file, then 
read and displayed each line in the file one by one, before automatically closing 
the file (thanks to the fact that open is a context manager): 

with open('todos.txt') as tasks: 
for chore in tasks: 

print(chore, end='') 


Y^uir -Pivst—evev 

Vrth" statement 

(bov\ro\wed -fvorn 
Chapteir t>). 

J 


Leti take another look at this with statement, highlighting where dunder en ter, 
dunder exit, and dunder init are invoked. WeVe numbered each of the 
annotations to help you understand the order the dunders exeeute in. Note that 
we doni see the initialization, setup, or teardown code here; we just know (and 
trust) that those methods run “behind the scenes” when needed: 



|. IVhen 
'm-tev-pvetev 
ewtouwievs -tbis 
Vrtb" statement, 
i-t be<\'ms bv tallm^ 

i J 1 i” 

duv\dcv 

assotia-tcd wi-bk i\\t 

tali bo u oycy\ . 


Z / S So T as "'"i" encuies, the 

r alU dunde " " e ^" that 

"tasks\ a * iabl e dalL9 ^ Wil ' be assi 9 ned J° * he 

with open('todos.txt’) as tasks: 
for chore in tasks: 

print(chore, end=’’) 

tbe towfce*t s dmder to Y ? ^ |S 

ey.aw.ple, tbe mteirwter ewsuv-es Ibat tbe ? 
tlosed pvopevly beWe towtmwm^- 

Wbafs required frow you 



Before we get to creating our very own context manager (with the help of a new class), leti 
review what the context management protocol expects you to provide in order to hook into 
the with statement. You must create a class that provides: 


1. an init method to perform initialization (if needed); 

2. an_ enter _method to do any setup; and 

3. an exit method to do any teardown (a.k.a. tidying-up). 

Armed with this knowledge, leti now create a context manager class, writing these methods 
one by one, while borrowing from our existing database code as needed. 
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implementing the protocol 


Create a New Context Manager Class 

To get going, we need to give our new class a name. Additionally, lef s put our new 
class code into its own file, so that we can easily reuse it (remember: when you put 
Python code in a separate file it becomes a module, which can be imported into other 
Python programs as required). 

Lefs call our new file DBcm. py (short for database context manager ), and lefs call our 
new class UseDatabase. Be sure to create the DBcm. py file in the same folder that 
currently contains your webapp code, as if s your webapp that’s going to import the 
UseDatabase class (once you Ve written it, that is). 

Using your favori te editor (or IDLE), create a new edit window, and then save the new, 
empty file as DBcm. py. We know that in order for our class to conform to the context 
management protocol it has to: 


1. 

provide an 

init 

2. 

provide an 

enter 

3. 

provide an 

exit 


method that performs initialization; 

_ method that includes any setup code; and 
method that includes any teardown code. 


Rementher: 
use CamelCase 
when naming 
a class in 
Python. 


For now, lef s add three “empty” definitions for each of these required methods to our 
class code. An empty method contains a single pass statement. Here’s the code so far: 


TWis is Yiha-t ouv 

looks likc 'm iPL-t* 
j\{, "the momeirrt, 

vt's rnade IA? 

3 sm^lc im^oY“t 
s-tatement 
{joap bhev- wth 
a tlass called 
u UsePa'tabase ,, 
-tha-t dontam^ 
-bhvee “emfty 
wcthods- 


• # • DBcm.py - /Users/paul/Desktop/NewBook/ch09/webapp/DBcm.py (3.5.1) 


import mysql.connector 

class UseDatabase: 

def 

init 

(self): 


pass 


def 

enter 

(self): 


pass 


def 

exit 

(self): 


pass 



Ln: 1 4 Coi: 0 


Note how at the top of the DBCm. py file weVe included an import statement, which 
includes the MySQL Connector functionality (which our new class depends on). 

All we have to do now is move the relevant bits from the log_request function 
into the correct method within the UseDatabase class. Well...when we say we, we 
actually mean you. If s time to roll up your sleeves and write some method code. 
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the context management protocol 


Initialize the Class with the Patabase Conf ig 


Let 5 s remind ourselves of how we intend to use the UseDatabase context manager. 
Here 5 s the code from the last chapter, rewritten to use a with statement, which itself 
uses the UseDatabase context manager that you’re about to write: 


from DBcm import UseDatabase 


Weves -£he da-fcabase 

^hamadtcHs-tids. 


Impovt “the don*tc*t 
ma^ayv- fv-or* -the 

W PBdw.p/ f »!«• 


dbconfig = { 'host': '127.0.0.1', 

'user': 'vsearch', 

'password': 'vsearchpasswd', 

'database': 'vsearchlogDB', } 



The doivfcex-1 
•^ahageir \re-tuv-hs 
a 'duJrsov”. 


The “UsePatabase 

mana^ev- e^edts 
{o v-edcWe a difctionairy 
of database dovmedW 

dbavad-tens-ttds. 


with UseDatabase(dbconfig) as cursor: 

_SQL = """insert into log 

(phrase, letters, ip, browser_string, resuits) 
values 

(O. _ O. _ O. — O,— O, _ \ IV ff IV 

■oS, o S , "o S , "6S , 

cursor.exeeute( SQL, (req.form['phrase'], 



Sharpen your pencil 


Lefs start with the_ init _method, which we'll use to initialize any 

attributes in the UseDataBase class. Based on the usage shown above, 
the dunder init method accepts a single argument, which is a dictionary 
of connection characteristics called config (which you'll need to add to 
the def line below). Lefs arrange for config to be saved as an attribute 
called conf iguration. Add the code required to save the dictionary to 
the conf iguration attribute to dunder init's code: 


import mysql.connector 


class UseDatabase: 



Comple-te -the 
'dc( u line. 


def 


init (self. 


) ^ 


/s there 
dhy-fchihg 
missihg 
•f *on\ heve? 


Save -the dohfijjuvatioh 
did-tiohavy -feo ah atbribu-te 
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dunder init done 


^ ■ J W WP ■ 'W■ ■ 

v Solution 


You started with the_ init _method, which was to initialize any 

attributes in the UseDataBase class.The dunder init method accepts 
a single argument, which is a dictionary of connection characteristics 
called conf ig (which you needed to add to the def line below). 

You were to arrange for conf ig to be saved to an attribute called 
conf iguration. You were to add the code required to save the 
dictionary to the conf iguration attribute in dunder init's code: 


import mysql.connector 


class UseDatabase: 



Dwndev "init" addepts a 
single didtionavy, whidh 
v/eVe dalling "dontig". 


def 


init (self. 


£orrfi<y did*t 


The value of the "don-fig" avgument 

u is as f9 hed attvibute dalled ** 
don+iguvation". Did you vemembev 
to pve-rix the attvibute with "selt"? 


AOY\ 


) —> bloY\C : 

r 

The (optional) "No*'® annotat' 1 
dontivms that this method has w> 
vetuvn value (whidh is nide to k ; ow), and 
the dolon tevminates the de-V me ' 


Your cowtext wawager begiws to take shape 

With the dunder init method written, you can move on to coding the dunder enter 

method (_enter_). Before you do, make sure the code you Ve written so far 

matches ours, which is shown below in IDLE: 


Make suve 

VOUV dundev- 

U' .1» 

mit 

ouvs. 



• DBcm.py - /Users/paul/Desktop/_NewBook/ch09/webapp/DBcm.py (3.5.1) 

import mysql.connector 


class UseDatabase: 

def _init_ (self, config: dict) -> None: 

self.configuration = config 

def _enter_ (self) : 

pass 

def _exit_ (self) : 

pass 


Ln: 14 Coi: 0 
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the context management protocol 


Perform Setup with Punder "enter" 

The dunder enter method provides a place for you to execute the setup code that 
needs to be executed before the suite in your with statement runs. Recall the code from 
the log_request function that handles this setup: 


tteves *tVic 
se-tup fcode 
-('vow' “the 



dbconfig = { 'host': '127.0.0.1', 

'user': 'vsearch', 

'password': 'vsearchpasswd', 

'database': 'vsearchlogDB', } 



SQL = """insert into log 

(phrase, letters, ip, browser string, results) 


This setup code uses the connection characteristics dictionary to connect to MySQL, 
then creates a database cursor on the connection (which we’11 need to send commands 
to the database from our Python code). As this setup code is something you’11 do every 
time you write code to talk to your database, let’s do this work in your context manager 
class instead so that you can more easily reuse it. 



The dunder enter method (_enter_) needs to use the configuration 

characteristics stored in self . configuration to connect to the 
database and create a cursor. Other than the mandatory self argument, 
dunder enter takes no other arguments, but needs to return the cursor. 
Complete the code for the method below: 



C^h you 
■fchihk o-f ah 

appvopHa-te 

dhho-ta-tioh? 


Vont -fovgci -fco 

v-e-tuvh ‘the £u\rsov. 



return 
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dunder enter done 


Mi 



Solutiori 


The dunder enter method (_enter_) uses the configuration 

characteristics stored in self . configuration to connect to the 
database and create a cursor. Other than the mandatory self argument, 
dunder enter takes no other arguments, but needs to return the cursor. 
You were to complete the code for the method below: 


pid you v-emc^bev- *fco 

Dye-fi* all atbribu^cs 

Ms/\-bV\ w sd*C w ? 



def enter (self) 


—> 'duvso/ 


sd-Pdorm — 

* sd-Pduv-sor — sel-p.dorm.dursov-O ^ 

return sd^.du^sov* g c S uVC ko rt$€r *to 

Vltaonfi^uration 
hev-e as opposed “to 

“dbdoyvfi^ . 


Retuvn the duv-sov. 



telis usevs 
o-f -fchis 

^lass what 
they eah 

e*pe£-fc -to 
be v-etuirKed 
*Prorio this 

method. 


PowY forget to pref ix all attributes wlth "self" 

You may be surprised that we designated conn and cursor as attributes in dunder 
enter (by prefixing each with self). We did this in order to ensure both conn and 

cursor survive when the method ends, as both variables are needed in the_ 

exit_method. To ensure this happens, we added the self prefix to both the conn 

and cursor variables; doing so adds them to the class 5 s attribute list. 

Before you get to writing dunder exit, confirm that your code matehes ours: 


# # # DBcm.py - /Users/paul/Desktop/_NewBook/ch09/webapp/DBcm.py (3.5.1 )* 

import mysql.connector 


class UseDatabase: 

def _init_ (self, config: dict) -> None: 

self.configuration = config 


v/ouVc *eavly done- 

0dy OY\t n\orC 

method “to wvrtc- 



def _enter_ (self) -> 'cursor' : 

self.conn = mysql.connector.connect(**self.configuration) 
self.cursor = self.conn.cursor() 
return self.cursor 

def _exit_ (self) : 

pass 


Ln: 1 6 Coi: 0 
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the context management protocol 


Perforw Teardown with Punder "exit" 


The dunder exit method provides a place for you to execute the teardown code 
that needs to be run when your with statement terminates. Recall the code from the 
log re que st function that handles teardown: 


cursor.execute( 


conn.commit() 
cursor.close() 
conn.close() 


SQL, (req.form['phrase’], 
req.form['letters'], 
req.remote_addr, 
req.user_agent.browser, 
res, )) 



TWis is “t 
itardow dodc- 


The teardown code commits any data to the database, then closes the cursor and the 
connection. This teardown happens every time you interact with the database, so let 5 s 
add this code to your context manager class by moving these three lines into dunder 
exit. 


Before you do this, however, you need to know that there’s a complication with dunder 
exit, which has to do with handling any exceptions that might occur within the 

with’s suite. When something goes wrong, the interpreter always notifies_exit_ 

by passing three arguments into the method: exec_type, exc_value, and exc_ 
trace. Your def line needs to take this into account, which is why weVe added the 
three arguments to the code below. Having said that, we’re going to ignore this exception- 
handling mechanism for now, but will return to it in a later chapter when we discuss 
what can go wrong and how you can handle it (so stay tuned). 



Add the 

-teavdow* 

dode Vicv^c- 


def 



The teardown code is where you do your tidying up. For this context 
manager, tidying up involves ensuring any data is committed to the 
database prior to closing both the cursor and the connection. Add the 
code you think you need to the method below. 

M *** *»<« ******* to, *ow 
exit (self, exc type, exc value, exc trace) : 
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dunder exit done 


v Solution 


The teardown code is where you do your tidying up. For this context 
manager, tidying up involves ensuring any data is committed to the 
database prior to closing both the cursor and the connection. You were to 
add the code you think you need to the method below. 


M ^ ^ese avguiweiyb (or »ow 

def exit (self, exc type, exc value, exc trace) —> Uoy\C : 


The pveviously saved attributes 

av-e used to dorv.rv.it ur. saved data, 

well as dlose the duirsov* and 

tormedtio*. As always, \revv.erv.bev* to 

J**™ y<*r attribute ha^es with 
w selt w . 



sel-f • dohh. dorv.rv.itO 


sel-PdursordoseO 


sel-Pdorm.dloseO 


T 

This ahho-btioh don-fivms ihai 
his method has ho vetuvh value; 
Sudh SnM-hiioM are optional 
but are good pvad-tide.. 


Your context manager is ready for testing 

With the dunder exit code written, it 5 s now time to test your context manager prior 
to integrating it into your webapp code. As has been our custom, weTl first test this 
new code at Python’s shell prompt (the >>>). Before doing this, perform one last check 
to ensure your code is the same as ours: 


The 

dorv.pl eted 

“UsePatabase’ 

dov\te*t 

mar^ev tlass 



# # # _ DBcm.py - /Users/paul/Desktop/_NewBook/ch09/webapp/DBcm.py (3.5.1) 

import mysql.connector 


class UseDatabase: 

def _init_ (self, config: dict) -> None: 

self.configuration = config 

def _enter_ (self) -> 'cursor 1 : 

self.conn = mysql.connector.connect(**self.configuration) 
self.cursor = self.conn.cursor() 
return self.cursor 


def _exit_ (self, exc_type, exc_value, exc_trace) 

self.conn.commit() 
self.cursor.close() 
self.conn.close() 


-> None 



Ln: 1 8 Coi: 0 


A "veal" dlass would 'mdlude dodume*taW, Wt ve ve 

* U. tW «d. t» «« - ** <» W’ 

) k I I a I J « i au.ia.AW.tC 
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Tqst DriI/q 


t “the 

ma^ager 
tlass -fvom 

*tV>c 

module file- 


the code for DBcm. py in an IDLE edit window, press F5 to test your context manager: 


Mse the 
£ohtex-fc 
^ahager to 
sehd some 
SQL- to the 

server a*d 
get some 
data baek. 


Python 3.5.1 Shell 


»> 

»> from DBcm import UseDatabase 

»> 

»> dbconfig = { 'host' : '127.0.0.1', 

'user' : 'vsearch' , 

'password' : 'vsearchpasswd' , 

'database' : 'vsearchlogDB' , } 

»> 

»> with UseDatabase (dbconfig) as cursor: 

_SQL = """Show tables""" 
cursor.exeeute(_SQL) 
data = cursor.fetchall() 






Ln: 45 Coi: 4 


Put the 
^ohhedtioh 
^haraeteristies 
a dietiohary. 



tuoles, wfch eatV, t« ? le toms ? owim$ b> a w». 


Theres wot wuch code here, is tbere? 

Hopefully, you’re looking at the code above and deciding there’s not an awful lot 
to it. As youVe successfully moved some of your database handling code into the 
UseDatabase class, the initialization, setup, and teardown are now handled “behind 
the scenes” by your context manager. Alt you have to do is provide the connection 
characteristics and the SQL query you wish to exeeute—the context manager does 
all the rest. Your setup and teardown code is reused as part of the context manager. 

It 5 s also clearer what the “meat” of this code is: getting data from the database and 
Processing it. The context manager hides the details of connecting/disconnecting to/ 
from the database (which are always going to be the same), thereby leaving you free to 
concentrate on what you’re trying to do with your data. 


Let’s update your webapp to use your context manager. 
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update your webapp 


Recowsidermg Your Webapp Code, 1 of 2 

It 5 s been quite a while since youVe considered your webapp’s code. 

The last time you worked on it (in Ghapter 7), you updated the log_request 
function to save the webapp’s web request to the MySQL database. The reason we 
started down the path to learning about classes (in Ghapter 8) was to determine the 
best way to share the database code you added to log_request. We now know 
that the best way (for this situation) is to use the just-written Use Databas e context 
manager class. 

In addition to amending log_request to use the context manager, the other 
function in the code that we need to amend work with the data in the database is 
called view_the_log (which currently works with the vsearch . log text file). 
Before we get to amending both of these functions, let’s remind ourselves of the 
current state of the webapp’s code (on this page and the next). WeVe highlighted the 
bits that need to be worked on: 


Your webajjjj’s 
code is in the 
vsearcli4wel).m 
file in your 
M Vehant»” f older. 


from flask import Flask, render_template, request, escape 
from vsearch import search41etters 


import mysql.connector 
app = Flask( name ) 


We need io 
^roiri u pfW' 

here ihs-fcead. 


This tode has -to 
be amended bo use 
•the w UsePatabase w 
manarer. 



def log_request(req: 'flask_request', res: str) -> None: 
"""Log details of the web request and the resuits.""" 


'user': 'vsearch', 

'password': 'vsearchpasswd', 

'database': 'vsearchlogDB', } 

conn = mysql.connector.connect(**dbconfig) 
cursor = conn.cursor() 

_SQL = """insert into log 

(phrase, letters, ip, browser_string, results) 
values 

( o, _ o, _ o, _ g,— g, — \ it ii ii 

"oS , "6S , "6S , "oS , "oS j 

cursor.execute(_SQL, (req.form ['phrase'], 

req.form['letters'], 
req.remote_addr, 
req.user_agent.browser, 
res, )) 


conn.commit() 
cursor.close() 
conn.close() 
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Recowsidermg Your Webapp Code, 2 of 2 


@app.route('/search4', methods=['POST']) 
def do_search() -> ! html': 

"""Extract the posted data; perform the search; return results.""" 

phrase = request.form['phrase'] 

letters = request.form['letters' ] 

title = 'Here are your results:' 

results = str(search41etters(phrase, letters)) 

log_request(request, results) 

return render_template('results.html ', 

the_title=title, 
the_j?hrase=phrase, 
the_letters=letters, 
the_results=results,) 

@app.route('/') 

@app.route('/entry') 

def entry_page() -> 'html': 

"""Display this webapp's HTML form.""" 
return render_template('entry.html ', 

the title='Welcome to search41etters on the web!') 


@app.route('/viewlog') 

def view_the_log() -> 'html': 

"""Display the contents of the log file as a HTML table.""" 
contents = [] 

with open('vsearch.log') as log: 
for line in log: 

contents.append([]) 
for item in line.split('|'): 

contents[-1].append(escape(item)) 


tities = ('Form Data ', 'Remote_addr ', 'User_agent' 

return render_template('viewlog.html ', 

the_title='View Log' , 
the_row_titles=tities , 
the data=contents,) 


This Co de io be arne ^ ed 

to use ih e data i* the 

database via £he "WseDaiabase" 
^ohtcxt mahagcv. 


Results') 


if _name_ == '_main_ 

app.run(debug=True) 
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updating logjrequest 


Recalliwg the "logjrequest" Function 

When it comes to amending the log_request function to use the Use Databas e 
context manager, a lot of the work has already been done for you (as we showed you 
the code we were shooting for earlier). 

Take a look at log_request once more. At the moment, the database connection 
characteristics dictionary (dbconf ig in the code) is defined within log_request. 
As you’11 want to use this dictionary in the other function you have to amend (view_ 
the_log), let’s move it out of the log_request 5 s function so that you can share it 
with other functions as needed: 


def log request(req: 'flask request', res: str) -> None: 


LeVs move this 

ci -the WW 

SO «t £3» be 

shavcd wth 
othev WW* 
as ve^Aed- 



cursor = conn.cursor() 

_SQL = """insert into log 

(phrase, letters, ip, browser_string, resuits) 
values 


(O. _ O, _ Q__ Q,_ Q, —, \ II !| II 

■oS, '65 ^ "65 ^ 65) 

cursor.exeeute(_SQL, (req.form['phrase 1 ], 

req.form['letters'], 
req.remote_addr, 
req.user_agent.browser, 
res, )) 

conn. commi t () 
cursor.close() 
conn.close() 


Rather than move dbconf ig into our webapp’s global space, it would be useful if we 
could somehow add it to our webapp’s internal configuration. 

As luck would have it, Flask (like many other web frameworks) comes with a built-in 
configuration mechanism: a dictionary (which Flask calls app . conf ig) allows you to 
adjust some of your webapp’s internal settings. As app . conf ig is a regular Python 
dictionary, you can add your own keys and values to it as needed, which is what youil 
do for the data in dbconf i g. 

The rest of log requesfs code can then be amended to use UseDatabase. 


Let’s make these changes now. 
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Awewdmg the "log Request" Fuwctiow 

Now that weVe applied the changes to our webapp, our code looks like this: 


■the old 


■fco -this 
fcpdated ohe. 




Wc adjusted 
•the dode -bo use 
“UsePatabase', 
beih^ suve *to pass 
m the da-tabase 

eoh-fioiuv-atioh 
fi u c- » 

4r\row\ aff toh-V^ • 


from DBcm import UseDatabase 
app = Flask(_name_) 


# # # vsearch4web.py - /Users/paul/Desktop/_NewBook/ch09/webapp/vsearch4web.py (3.5.1) 

from flask import Flask, render_template, request, escape 
from vsearch import search41etters 

We added the 

^Ohhe^tioh 

^hav-a^tevisties 
difi-tiohary -to the 

app. conf ig [ 1 dbconf ig' ] = {'host': '127.0.0.1', s ^wtijuvatioh. 

1 user' : 'vsearch' , I 

1 password’ : 'vsearchpasswd' , s J 
'database': 'vsearchlogDB' , } 


def log_request (req: 'flask_request’ , res: str) -> None: 
" ,M, Log details of the web request and the results.""" 


with UseDatabase(app.config[ 'dbconfig' ]) as cursor: 

_SQL = """insert into log 

(phrase, letters, ip, browser_string, results) 
values 

(%s, %s, %s, %s, %s)""" 

cursor.execute(_SQL, (req.form[ 'phrase' ], 

req.form[ 'letters' ], 
req.remote_addr, 
req.user_agent.browser , 
res, )) 


Ln: 1 0 Coi: 0 


Near the top of the file, weVe replaced the import mysql. connector statement 
with an import statement that grabs UseDatabase from our DBcm module. The 
DBcm. py file itself includes the import mysql. connector statement in its code, 
hence the removal of import mysql .connector from this file (as we don’t want 
to import it twice). 

WeVe also moved the database connection characteristics dictionary into our webapp’s 
configuration. And weVe amended log_request 5 s code to use our context manager. 

After all your work on classes and context managers, you should be able to read and 
understand the code shown above. 

Let 5 s now move onto amending the view_the_log function. Make sure your 
webapp code is amended to be exactly like ours above before turning the page. 


you are here ► 


351 








updating viewjthejog 

Recalling the "view_theJog" FuhcHoh 

Let 5 s take a long, hard look at the code in view_the_log, as it’s been quite a while 
since youVe considered it in detail. To recap, the current version of this function 
extracts the logged data from the vsearch.log text file, turns it into a list of lists 
(called contents), and then sends the data to a template called viewlog . html: 

@app.route('/viewlog') 
def view_the_log() -> 'html': 

contents = [] 

with open('vsearch.log') as log: 

eseaped items, whieh^.- for ii ne i n log: 

ave appended to “the contents . append ([ ]) 

for item in line. split (' | ') : 

contents[-1].append(escape(item)) 

tities = ('Form Data', ' Remote_addr', 'User_agent', 'Results') 

return render_template('viewlog.html', 

the_title='View Log’ , 

The pvodessed log the_row_titles=tities, 

data is se*t to the the_data=contents,) 

template -Pov display. 

Here’s what the output looks like when the viewlog . html template is rendered with 
the data from the contents list of lists. This functionality is currently available to 
your webapp via the /viewlog URL: 


£jv~ab eaeh Ime o£ 
data £vom the -f ’»le, 

and then tvanstovn» 

it 'mto a list ot 


n 


O 127.0.0.1:5000/viewlog 


The data £vom 
w tonte*ts ’ is displayed 
in the -fovm. Note - 
hov/ the -fovm 

data ( w phvase w and 

w lettevsO is pvesented 
in a smgle tolumn. 


View Log 


Form Data 



ImmutableMultiDictCfClettersl 
'aeiou'), ('phrase', 'hitch- 
hiker')]) 


ImmutableMultiDict([('letters', 


Remote_addr User_agent 


127.0.0.1 


Mozilla/5.0 
(Macintosh; Intel 
Mac OS X 10_11_2) 
AppleWebKit/601.3.9 i {'e', 'i'} 
(KHTML, like Gecko) 
Version/9.0.2 
Safari/601.3.9 



Mozilla/5.0 
(Macintosh; Intel 
Mac OS X 10_11_2) 


{'e' 'u' 
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Ifs Not Just the Code That Chawges 

Before diving in and changing the code in view_the_log to use your context 
manager, lefs pause to consider the data as stored in the log table in your database. 
When you tested your initial log_request code in Ghapter 7, you were able to 
log into the MySQL console, then check that the data was saved. Recall this MySQL 
console session from earlier: 


File Edit" Window Help Checking our log DB 


$ mysql -u vsearch -p vsearchlogDB 
Enter password: 

Welcome to MySQL monitor... 


mysql> select * from log; 
+-+- 


id 

1 

-X- 

ts 


1 

_ _L- 

phrase 

-+- 

1 

1 

letters 

1 

ip 

1 

browser string 

1 

results 


1 

"T" 

1 

2016-03-09 

13:40:46 

1 

1 

life, the uni 

T 

... ything | 

1 

aeiou 

1 

1 

127.0.0.1 

1 

1 

firefox 

1 

1 

{'u', 'e' 

, 'i', 'a'} 

2 

1 

2016-03-09 

13:42:07 

1 

hitch-hiker 

aeiou 

1 

127.0.0.1 

1 

safari 

1 

{'i', 'e' 

} 

3 

1 

2016-03-09 

13:42:15 


galaxy 

1 

xyz 


127.0.0.1 

1 

chrome 

1 

{'y', 'x' 

} 

4 

1 

2016-03-09 

13:43:07 

1 

hitch-hiker 

1 

xyz 

1 

127.0.0.1 

1 

firefox 

1 

set () 

— 


— 

— 


— 

-+- 


— 


— 


— 



4 rows in set (0.0 sec) 

mysql> quit 
Bye 


If you consider the above data in relation to whafs currently stored in the vsearch . 
log file, ifs ciear that some of the processing view_the_log does is no longer 
needed, as the data is now stored in a table. Here’s a snippet of what the log data looks 
like in the vsearch . log file: 


\ 

The log data saved 
m a database table 


ImmutableMultiDict([('phrase ', 'galaxy'), ( 'letters ', 'xyz')]) |127.0.0.1|Mozilla/5.0 (Macintosh; Intel 

Mac OS X 10 11 2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36|{'x ', 1 y 1 } 


Some of the code currently in view_the_log is only there because the log data 
is currently stored as a collection of long strings (delimited by vertical bars) in the 
vsearch.log file. That format worked, but we did need to write extra code to make 
sense of it. 

This is not the case with data in the log table, as it is “structured by default.” This 
should mean you don’t need to perform any additional processing within view_ 
the_log: all you have to do is extract the data from the table, which—happily—is 
returned to you as a list of tuples (thanks to DB-APFs f etchall method). 

On top of this, the data in the log table separates the value for phrase from the 
value for letters. If you make a small change to your template-rendering code, the 
output produced can display five columns of data (as opposed to the current four), 
making what the browser displays even more useful and easier to read. 


) 

U>e log data saved as 
o*e lohg sfe-inj i h -the 
vse3\r£h*lo0 -pile. 
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getting to view thelog 

Awewdmg the 'VievvTheJog" Fuwctiow 

Based on everything discussed on the last few pages, youVe two things to do to amend 
your current view_the_log code: 


1. Grab the log data from the database table (as opposed to the file). 

2. Adjust the tities list to support five columns (as opposed to four). 

If you’re scratching your head and wondering why this small list of amendments 
doesn’t include adjusting the viewlog . html template, wonder no more: you don’t 
need to make any changes to that file, as the current template quite happily processes 
any number of tities and any amount of data you send to it. 

Here’s the view_the_log function’s current code, which you are about to amend: 


/\s a vcsul-t o-f 
iask #1 above, 
*tbis Codt 
nec ds *bo be 
v-cfladcd* 


As a 

o? task #2- 
afeove» tK» s 
Ime *eeds lo 
fee a*»>e"ded- 


as log: 


@app.route(’/viewlog’) 
def view_the_log() -> ’html’ 

f contents = [] 

with open('vsearch.log') 
for line in log: 

contents.append([]) 
for item in line.split('|'): 

contents[-1].append(escape(item)) 

tities = ('Form Data', 'Remote_addr', 'User_agent', 

return render_template(’viewlog.html', 

the_title='View Log' , 
the_row_titles=tities, 
the data=contents,) 



Here^s the SCII query youMI need 


Ahead of the next exercise (where you’11 update the view_the_log function), here 5 s 
an SQL query that, when exeeuted, returns all the logged data stored in the webapp’s 
MySQL database. The data is returned to your Python code from the database as a list 
of tuples. Youdl need to use this query in the exercise on the next page: 


select phrase, letters, ip, browser_string, resuits 
from log 


’ Results') 
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■^grr r r 


X 


Here's the view_the_log function, which has to be amended to use 
the data in the log table. Your job is to provide the missing code. Be sure 
to read the annotations for hints on what you need to do: 


@app.route('/viewlog') 

def view the log() -> 'html': 


with 


Use youv wana^cv hcv"e> 

and doni hy-ojei the twvsov. 


SQL = MMM select phrase, letters, ip, browser_string, results 
from log""" 


hifch to \umn 
tlcs ave 
iss'm<\ *fvow» 
:ve? 



'Remote addr', 


return render_template('viewlog.html', 

the_title='View Log', 
the_row_titles=titles, 
the data=contents,) 


Send ihe ^evy io ihe sev -ven, 
ihen teteh ih e results. 

'User agent', 'Results') 




Yep—that was our goal ali along. 

By moving the log data into a MySQL database, 
you Ve removed the requirement to create, and then 
process, a custom text-based file format. 

Also, by reusing your context manager, you Ve 
simplified your interactions with MySQL when 
working in Python. What 5 s not to like? 
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view_the_log done 


^ | J | ■ ^ ■ 

v Solution 


Here's the view_the_log function, which has to be amended to use 
the data in the log table. Your job was to provide the missing code. 


Add m ihe 

dovvcd-1 


n , /, / , N TU\ S is ”tV>c l*mc todc 

@app.route('/viewlog') 1 1 u p 

def view_the_log () -> 'html' : kcm *bbe l°9uJ rC< V ACS ^' ,on 

with UscPatabasc^app doh-fi^C^bdoh-fi^J) as duv-sov-: 

_SQL = """select phrase, letters, ip, browser_string, results 
from log""" 

durso^e%e£ute(_S$L) f < \ uc<r y to ihe sevvev, -the* 

Co ^ e ^ tl^:h^ al10 J of ihe irfSd^dafat' 

tities = ( ‘Phrase’ , Letters’ , 'Remote_addr', 'User_agent' , 'Results') 

return render_template('viewlog.html', 

the_title='View Log', 
the_row_titles=titles, 
the data=contents,) 


lt'$ nearly time for owe last Test Prive 

Before taking this new version of your webapp for a spin, take a moment to 
confirm that your view_the_log function is the same as ours: 


• 0 • vsearch4web.py -/Users/paul/Desktop/NewBook/ch09/webapp/vsearch4web.py (3.5.1) 

@app.route( '/viewlog' ) 

def view_the_log () -> 'html': 

11 ""Display the contents of the log file as a HTML table.""" 
with UseDatabase(app.config[ 'dbconfig' ]) as cursor: 

_SQL = """select phrase, letters, ip, browser_string, results 

from log""" 
cursor.execute(_SQL) 
contents = cursor.fetchall() 

tities = ('Phrase', 'Letters', 'Remote_addr' , 'User_agent' , 'Results') 
return render_template( 'viewlog.html' , 

the_title=' View Log' , 
the_row_titles=tities, 
the data=contents,) 


Ln: 1 Coi: 0 



356 Chapter 9 
















the context management protocol 


Tqst DriVq- 

lt’s time to take your database-ready webapp for a spin. 

Be sure the DBcm. py file is in the same folder as your vsearch4web. py file, then start your 
webapp in the usual way on your operating system: 


• Use python3 vsearch4web . py on LinuxIMac OS X 

• Use py -3 vsearch4web. py on Windows. 

Use your browser to go to your webapp’s home page (running at http://127.0.0.1:5000 ), then enter a 
handful of searches. Once you’ve confirmed that the search feature is working, use the 
/viewlog URL to view the contents of your log in your browser window. 

Although the searches you enter will very likely differ from ours, here’s what we saw in our browser 
window, which confirms that everything is working as expected: 


< | > | HJ to 127.0.0.1:5000/viewlog C (h 0 _|_ 


View Log 


Phrase 

Letters 

Remote_addr 

User_agent 

Resui ts 

life, the universe, and everything 

aeiou 

127.0.0.1 

firefox 

{'u', 'e’, V, 
'a'} 

hitch-hiker 

aeiou 

127.0.0.1 

safari 

{V, 'e'} 

galaxy 

xyz 

127.0.0.1 

chrome 

{V, v> 

hitch-hiker 

xyz 

127.0.0.1 

firefox 

set() 

lightning in a bottle 

aeiou 

127.0.0.1 

firefox 

{V, 'a', 'o', 
'e'} 

testing the database-enabled 
webapp 

aeiou 

127.0.0.1 

firefox 

{-e 1 , 'a', 'i'} 


This browser output confirms the logged data is being read from the MySQL database when the 
/viewlog URL is accessed. This means the code in view_the_log is working—which, incidentally, 
confirms the log_request function is working as expected, too, as it’s putting the log data in the 
database as a resuit of every successful search. 

Only if you feel the need, take a few moments to log into your MySQL database using the MySQL 
console to confirm that the data is safely stored in your database server. (Or just trust us: based on 
what our webapp is displaying above, it is.) 
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answer those questions 


AII That Remains... 

It’s now time to return to the questions first posed in Chapter 7: 

• How many requests have been responded to? 

• WhaVs the most common list of letters? 

• Which IP addresses are the requests comingfrom? 

• Which browser is being used the most? 


Although it is possible to write Python code to answer these questions, we aren’t going 
to in this case, even though weVe just spent this and the previous two chapters looking 
at how Python and databases work together. In our opinion, creating Python code to 
answer these types of questions is nearly always a bad move... 



So if I'm not going to use Python to 
answer these questions, what should 
I use instead? I learned a little bit 
about databases and SQL while working 
through Chapter 7—would SQL queries 
be a good fit here? 


SQL is definitely the way to go. 

These types of “data questions” are best answered 
by your database technology’s querying mechanism 
(which, in MySQL, is SQL). As you’11 see on the 
next page, you’d be unlikely to produce Python code 
as quickly as writing the SQL queries you need. 

Knowing when to use Python and when not to is 
important, as is knowing what sets Python apart 
from many other programming technologies. While 
most mainstream languages support classes and 
objects, few provide anything close to Python’s 
context management protocol. (In the next chapter, 
you’11 meet another feature that sets Python apart 
from many other languages: function decorators.) 

Before we get to the next chapter, let’s take a quick 
(one page) look at those SQL queries... 
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Answering the Pata Questione 

Let’s take the questions first posed in Chapter 7 one by one, answering each with the 
help of some database queries written in SQL. 


How mawy requests have beew respowded to? 


If you’re already a SQL dude (or dudette), you may be scoffing at this question, 
seeing as it doesn’t really get much simpler. You already know that this most basic of 
SQL queries displays all the data in a database table: 


select * from log; 

To transform this query into one that reports how many rows of data a table has, pass 
the * into the SQL function count, as follows: 

select count(*) from log; <=- 


We ^y\o- sKowihg you 
■the ahswew hev-e. If y ou 

to see them, you'l| have 
to ruo these ^uev-ies yowvsel-f 
ih the My£QL dowole (see 
Chaptev 7 for a vetv-esh). 


What's the most commori list of letters? 

The SQL query that answers this question looks a little scary, but isn’t really Here it 
is: 

select count(letters) as 'count', letters 
from log 

group by letters 
order by count desc 
limit 1; 

Which IF addresses are the requests comiwg from? 

The SQL dudes/dudettes out there are probably thinking “that’s almost too easy”: 

select distinet ip from log; 

Which browser is bemg used the most? 

The SQL query that answers this question is a slight variation on the query that 
answered the second question: 

select browser_string, count(browser_string) 
from log 

group by browser_string 
order by count desc 
limit 1; 


/\s su^csicd m Cbaptev 
aUays vedommend tb* 
book whein somConC s Hvst, 

Icav^ma SQL (as as 

ufdatmj pv-evious knowled^e 

-tkat wiijkt be a bit v-us-ty • 




as 'count' 


So there you have it: all your pressing questions answered with a few simple SQL queries. 

Go ahead and try them at your mysql> prompt before starting in on the next chapter. 
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Chapter 9'$ Code, 1 of 2 


TWis is -the 

wanayr LoAc 
m "PBt*»•?/• 


TWis >s tV»e 
V»a\-f ’^' e 

Yieba?? tode >« # 
* vse a r eV*^«b.f/--v 


from flask import Flask, render_template, request, escape 
from vsearch import search41etters 

from DBcm import UseDatabase 

app = Flask( name ) 


app.config['dbconfig'] = {'host': '127.0.0.1', 

'user': 'vsearch', 

'password': 'vsearchpasswd', 

'database': 'vsearchlogDB', } 

def log_request(req: 'flask_request', res: str) -> None: 

with UseDatabase(app.config['dbconfig']) as cursor: 

_SQL = """insert into log 

(phrase, letters, ip, browser_string, results) 
values 

/9-a 9-0 9-0 9-0 9- O \ HI!!! 

V o O f OOf o o ^ o o ^ oo / 

cursor.execute(_SQL, (req.form['phrase'], 

req.form['letters'], 
req.remote_addr, 
req.user agent.browser, 
res, )) 



import mysql.connector 
class UseDatabase: 

def _init_(self, config: dict) -> None: 

self.configuration = config 

def _enter_(self) -> 'cursor': 

self.conn = mysql.connector.connect(**self.configuration) 
self.cursor = self.conn.cursor() 
return self.cursor 

def _exit_(self, exc_type, exc_value, exc_trace) -> None: 

self.conn.commit() 
self.cursor.close() 
self.conn.close() 
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Chapter 9'$ Code, 2 of 2 



This is the seto»d hal-f o-f the 
webap P £ode m "vseav-dh^eb.py". 


@app.route('/search4', methods=['POST']) 
def do_search() -> 'html': 

phrase = request.form['phrase'] 

letters = request.form['letters'] 

title = 'Here are your results:' 

results = str (search41etters(phrase, letters)) 

log_request(request, results) 

return render_template('results.html', 

the_title=title, 
the_phrase=phrase, 
the_letters=letters, 
the_results=results,) 

@app.route('/') 

@app.route('/entry') 

def entry_page() -> 'html': 

return render_template('entry.html', 

the_title='Welcome to search41etters on the web!') 

@app.route('/viewlog') 

def view_the_log() -> 'html': 

with UseDatabase(app.config['dbconfig']) as cursor: 

_SQL = """select phrase, letters, ip, browser_string, results 
from log""" 
cursor.execute(_SQL) 
contents = cursor.fetchall() 

tities = ('Phrase', 'Letters', 'Remote_addr', 'User_agent', 'Results') 

return render_template('viewlog.html', 

the_title='View Log', 
the_row_titles=titles, 
the_data=contents,) 

if _name_ = '_main_' : 

app.run(debug=True) 
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10 fonetion deco?ators 




When it comes to augmenting your code, Chapter 9’s context 
management protocol is notthe only game in town. 

Python also lets you use function decorators, a technique whereby you can add code 
to an existing function without having to change any of the existing function’s code. If 
you think this sounds like some sort of black art, don’t despair: it’s nothing of the sort. 
However, as coding techniques go, creating a function decorator is often considered to 
be on the harder side by many Python programmers, and thus is not used as often as 
it should be. In this chapter, our plan is to show you that, despite being an advanced 
technique, creating and using your own decorators is not that hard. 


this is a new chapter 
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pause for thought 


Your Webapp Is Workiwg Well, Put... 

YouVe shown the latest version of your webapp to a colleague, and theyVe 
impressed by what youVe done. However, they pose an interesting question: is it 
wise to let any web user view the log page? 

The point theyVe making is that anybody who is aware of the /viewlog URL can 
use it to view the logged data whether they have your permission or not. In fact, at 
the moment, every one of your webapp 5 s URLs are public, so any web user can 
access any of them. 

Depending on what you’re trying to do with your webapp, this may or may not 
be an issue. However, it is common for websites to require users to authenticate 
before certain content is made available to them. It 5 s probably a good idea to be 
prudent when it comes to providing access to the /viewlog URL. The question is: 
how doyou restrict access to certain pages in your webapp? 


Owly authenticated users gain access 


You typically need to provide an ID and password when you access a website 
that serves restricted content. If your ID/password combination match, access 
is granted, as youVe been authenticated. Once you’re authenticated, the System 
knows to let you access the restricted content. Maintaining this state (whether 
authenticated or not) seems like it might be as simple as setting a switch to True 
(access allowed; you are logged in) or False (access forbidden; you are not logged 
in). 


That sounds straightf orward to 
me. A simple HTML form can ask for the 
users credentials, and then a boolean on 
the server can be set to "True" or "False" as 
needed, right? 
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lt’s a bit more complicated than that. 

There 5 s a twist here (due to the way the Web works 
which makes this idea a tad more complicated 
than it at first appears. Let’s explore what this 
complication is first (and see how to deal with it 
before solving our restricted access issue. 



function decorators 


The Web Is Stateless 


In its most basic form, a web server appears incredibly silly: each and every 
request that a web server processes is treated as an independent request, 
having nothing whatsoever to do with what came before, nor what comes 
after. 

This means that sending three quick requests to a web server from your 
computer appears as three independent individual requests. This is in spite of 
the fact that the three requests originated from the same web browser running 
on the same computer, which is using the same unchanging IP address (which 
the web server sees as part of the request). 

As stated at the top of the page: it’s as if the web server is being silly. Even 
though we assume the three requests sent from our computer are related, the 
web server doesn’t see things this way: every web request is independent of what 
came before it , as well as what comes after. 


HTTP is to blawe... 

The reason web servers behave in this way is due to the protocol that 
underpins the Web, and which is used by both the web server and your web 
browser: HTTP (the HyperText Transfer Protocol). 

HTTP dictates that web servers must work as described above, and the reason 
for this has to do with performance: if the amount of work a web server 
needs to do is minimized, it 5 s possible to scale web servers to handle many, 
many requests. Higher performance is achieved at the expense of requiring 
the web server to maintain information on how a series of requests may be 
related. This information—known as state in HTTP (and not related to 
OOP in any way)—is of no interest to the web server, as every request is 
treated as an independent entity. In a way, the web server is optimized to 
respond quickly, but forget fast, and is said to operate in a stateless manner. 

Which is all well and good until such time as your webapp needs to remember 
something. 


When running as a web 
server, I pride myself in 
responding quickly...and 
forgetting fast. I'm stateless. 


0 






If only the Web were that simple. 

When your code is running as part of a web 
server, its behavior can differ from when you 
run it on your computer. Let’s explore this 
issue in more detail. 
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globals are problematic 


Your Web Server (Not Your Computer) Ruws 
Your Code 


When Flask runs your webapp on your computer, it keeps your code in memory at all 
times. With this in mind, recall these two lines from the bottom of your webapp’s code, 
which we initially discussed at the end of Ghapter 5: 

if name_ == '_main ' :_ This line o( tode does MOT exeduie 

a PP .run(debug=True) ^- whe* this tode is impoUed. 

This i f statement checks to see whether the interpreter is executing the code directly 
or whether the code is being imported (by the interpreter or by something like 
PythonAnywhere). When Flask executes on your computer, your webapp’s code runs 
directly, resulting in this app . run line executing. However, when a web server is 
configured to execute your code your webapp 5 s code is imported , and the app . run line 
does not run. 


Why? Because the web server runs your webapp code as it sees fit. This can involve the 
web server importing your webapp’s code, then calling its functions as needed, keeping 
your webapp 5 s code in memory at all times. Or the web server may decide to load/ 
unload your webapp code as needed, the assumption being that, during periods of 
inactivity, the web server will only load and run the code it needs. It’s this second mode 
of operation—where the web server loads your code as and when it needs it—that can 
lead to problems with storing your webapp 5 s state in variables. For instance, consider 
what would happen if you were to add this line of code to your webapp: 

logged_in = False 

if _name_ == '_main_ 

app.run(debug=True) 



The idea here is that other parts of your webapp can refer to the variable logged_ 
in in order to determine whether a user is authenticated. Additionally, your code 
can change this variable’s value as needed (based on, say, a successful login). As the 
logged_in variable is global in nature, all of your webapp’s code can access and set 
its value. This seems like a reasonable approach, but has two problems. 

Firstly, your web server can unload your webapp’s running code at any time (and 
without warning), so any values associated with global variables are likely lost, and 
are going to be reset to their starting value when your code is next imported. If a 
previously loaded function sets logged_in to True, your reimported code helpfully 
resets logged_in to False, and confusion reigns... 

Secondly, as it stands, there 5 s only a single copy of the global logged_in variable in 
your running code, which is fine if all you ever plan to have is a single user of your 
webapp (good luck with that). If you have two or more users each accessing and/or 
changing the value of logged in, not only will confusion reign, but frustration will 
make a guest appearance, too. As a general rule of thumb, storing your webapp’s state 
in a global variable is a bad idea. 



Don 1 t store 
your wetapp’s 
state in global 
variables. 
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lt'$ Time for a Pit of a Session 

As a resuit of what we learned on the last page, we need two things: 

• A way to store variables without resorting to using globals 

• A way to keep one webapp user’s data from interferring with another 5 s 

Most webapp development frameworks (including Flask) provide for both of these 
requirements using a single technology: the session. 

Think of a session as a layer of state spread on top of the stateless Web. 

By adding a small piece of identification data to your browser (a cookie ), and 
linking this to a small piece of identification data on the web server (the session 
/Z)), Flask uses its session technology to keep everything straight. Not only can you 
store state in your webapp that persists over time, but each user of your webapp 
gets their own copy of the state. Gonfusion and frustration are no more. 

To demonstrate how Flask 5 s session mechanism works, let’s take a look at a very 
small webapp that is saved to a file called quick_session . py. Take a moment 
to read the code first, paying particular attention to the highlighted parts. We’11 
discuss what 5 s going on after you Ve had a chance to read this code: 
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gotta love sessions 


Flask'$ Session Technology Adds State 


In order to use FlasUs session technology, you first have to import session from 
the f lask module, which the quick_session . py webapp you just saw does on 
its very first line. Think of session as a global Python dictionary within which you 
sto re your webapp’s state (albeit a dictionary with some added superpowers): 

S-fcairfc by 

from flask import Flask, session . , / 

iruPortiha 
« • » J 
sessioh . 



Even though your webapp is stili running on the stateless Web, this single import gives 
your webapp the ability to remember state. 

Flask ensures that any data stored in session exists for the entire time your webapp 
runs (no matter how many times your web server loads and reloads your webapp code). 
Additionally, any data stored in session is keyed by a unique browser cookie, which 
ensures your session data is kept away from that of every other user of your webapp. 

Just how Flask does all of this is not important: the fact that it does is. To enable all this 
extra goodness, you need to seed Flask’s cookie generation technology with a “secret 
key,” which is used by Flask to encrypt your cookie, protecting it from any prying eyes. 
Here 5 s how quick session.py does this: 


Finct out more about 
Flask sessions here: 
http:/ / llask.pocoo.org/ 
ctocs/ 0 . 11 / api/#sessions 



Cv-eate a 
plask webayy 
m “the usual way- 


app = Flask(_name ) 

app.secret key = 'YouWillNeverGuess 


Seed Flask’* tookie-wafa» 

lefcUoloy/ W.th a seftret k«Y- ( p« 

awy stvm^ ^ ^ ou ^ ', 

lilce a»y °thev passviovd you **> it 

should be havd -to yess.) 


FlasUs documentation suggests picking a secret key that is hard to guess, but any 
stringed value works here. Flask uses the string to encrypt your cookie prior to 
transmitting it to your browser. 


Once session is imported and the secret key set, you can use session in your 
code as you would any other Python dictionary. Within quick_session . py, the 
/setuser URL (and its associated setuser function) assigns a user-supplied value to the 
user key in session, then returns the value to your browser: 


J~be value o-f -the 

W w vaHable 

is assig^ed to 


the 


usev- 


w key 


,h the 'sessioh ;; 


dieti 


lohav-y. 


@app.route('/setuser/<user>') 
def setuser(user: str) -> str: 
session [' user' ] = user 
return 'User value set to: 



+ session['user'] 


The URL expeets to 
be pirovided with a 
^ value to assign to the 
"useir" vairiable (youll 
see how this wovks in 
a little bit) 


Now that weVe set some session data, let’s look at the code that accesses it. 
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Pictiowary Lookup Rctrieves State 


Now that a value is associated with the user key in session, it 5 s not hard to 
access the data associated with user when you need it. 


The second URL in the quick_session . py webapp, /getuser, is associated 
with the getuser function. When invoked, this function accesses the value 
associated with the user key and returns it to the waiting web browser as part 
of the stringed message. The getuser function is shown below, together with 
this webapp 5 s dunder name equals dunder main test (first discussed near the end of 
Ghapter 5): 


@app.route('/getuser') 
def getuser() -> str: 

return 'User value is currently set to: ' + session['user'] 


{ 


if 


name == '_main_ 

app.run(debug=True) 


As is the dustom with ali Flask apps, we 
iontrol when "app.vuh" exedutes wsihO 
this well-established Pythoh idiom. 



/\ttess'm$ the data m 
"session" is not havd- lt’s a 
dittionavy looki*p- 


Time for a Test Prive? 

It’s nearly time to take the quick_session . py webapp for a spin. However, 
before we do, let’s think a bit about what it is we want to test. 

For starters, we want to check that the webapp is storing and retrieving the session 
data provided to it. On top of that, we want to ensure that more than one user 
can interact with the webapp without stepping on any other user 5 s toes: the session 
data from one user shouldfft impact the data of any other. 

To perform these tests, we’re going to simulate multiple users by running multiple 
browsers. Although the browsers are all running on one computer, as far as the 
web server is concerned, they are all independent, individual connections: the 
Web is stateless, after all. If we were to repeat these tests on three physically 
different computers on three different networks, the results would be the same, 
as all web servers see each request in isolation, no matter where the request 
originates. Recall that the session technology in Flask layers a stateful 
technology on top of the stateless Web. 

To start this webapp, use this command within a terminal on Linux or Mac OS X: 

$ python3 quick session.py 


or use this command at a command prompt on Windows : 

C:\> py -3 quick session.py 
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setting session 



With the quick_session.py webapp up and running, let’s open a Chrome browser and use it 
to set a value for the user key in session. We do this by typing /setuser/Alice into the 
location bar, which instructs the webapp to use the value Alice for user: 



Appcrdhr^ a *ame 4 
the end the M 1 - 

tells «ebaf? t» 
wse "Alite” as the 
value tor user • 


127.0.0.1:5000/setusc X 


C 127.0.0.1:5000/setuser/Alice 


☆ 


User value set to: Alice 


V 


The webapp £ohfirms 
4at "Alide” is 

■the durrert value 

°( W ; 4r -this 
browser. 


Next, let’s open up the Opera browser and use it to set the value of user to Bob (if you don’t have 
access to Opera, use whichever browser is handy, as long as it’s not Chrome): 


Q localhost:5000/setuser/Bob 


© localhost:5000/setuser/Bob 


User value set to: Bob 

'N 


Cowf irmatior 
that Ver” has 
beer set 4 
w fcob” b7 the 
webapp 


fi 


As with Chrome, we 
apperd a rame to the 

URL to «t the value of 
useir ' I» this ease, we 
a PP e ^ the name "Bob". 
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When we opened up Safari (or you can use Edge if you are on Windows), we used the webapp’s other 
URL, /getuser, to retrieve the current value of user from the webapp. However, when we did this, 
we’re greeted with a rather intimidating error message: 



The Vyiuse/ 
URL le-b y°* 
cMttk ihe 
duv-v-eni value oi 


u w 

usev • 


127.0.0.1:5000/getuser 


c 


builtins.KeyError 


KeyError: 'user' 


Traceback (most recent call last) 


File 7Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site- 
packages/flask/app.py", line 1836 , in_call_ 

return self.wsgi_app(environ, start_response) 

File 7Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site- 
packages/flask/app.py", line 1820 , in wsgi_app 

response = self.make_response(self.handle_exception(e)) 

File 7Library/Frameworks/Python.franiework/Versions/3.5/lib/python3.5/site- 
packages/flask/app.py", line 1403 , in handle_exception 


JL _°L [+ 



Let’s use Safari to set the value of user to Chuck: 


• ® < ED v 127.0.0.1:5000/setuser/Chuck C> 


User value set to: Chuck 


Now ihai weve used Saiavi io 
sci a value -for w usev > ihe webapp 
happily v-esponds wiih a message 

ihai Thudk” has bee* 
added io ihe “sessio/ dieiionary. 





V^ikes/ Thais <^uiie ihe 
e\nro\r mcssagc, is* i 
ii? The irnpoviahi bii 
is at ihe iop- we have 
a u ^eyE\r\ro\r w , as we 
haveir/t used SaiaH io 
set a value iov “use/ 
yei- (Remembev*-' we 
sei a use\r value usirc^ 
Chvome a*d Opera, *ot 
Sa-Pari.) 
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getting session 



Now that we’ve used the three browsers to set values for user, let’s confirm that the webapp (thanks 
to our use of session) is stopping each browser’s value of user from interfering with any other 
browser’s data. Even though we’ve just used Safari to set the value of user to Chuck, let’s see 
what its value is in Opera by using the /getuser URL: 


Q localhost:5000/getuser 

C :: : © localhost:5000/getuser 




User value is currently set to: Bob 

Pespite the tadt that SaW. set 

"use/ to "Chudk", the 0\ccZ bvovisev 
£o »W that it stili thmks the value ot 


«use/ is "Bob". 


Having confirmed that Opera is showing users value as Bob, let’s return to the Chrome browser 
window and issue the /getuser URL there. As expected, Chrome confirms that, as far as it’s concerned, 
the value of user is Alice: 


As cxpcd-tcd, 

Chlrorne s-fcill 
■fchihks value 
"user" is 

"Micer 


G Google X Q 127.0.0.1:5000/getus x 


/ -' 

4* C D 127.0.0.1:5000/getuser 

— 

User value is currently set to: Alice 
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We’ve just used Opera and Chrome to access the value of user using the /getuser URL, which just 
leaves Safari. Here’s what we see when we issue /getuser in Safari, which doesn’t produce an error 
message this time, as user has a value associated with it now (so, no more KeyError): 


• • • < > [ 

□ v. 127.0.0.1:5000/getuser 

cl 

di 

S 



User value is currently set to: Chuck 



as it's tontevned— W 
value of "usev is stili 

"CWdk". 



No, not quite—it ali happens in the webapp. 

The use of the session dictionary in the webapp enables 
the behavior you’re seeing here. By automatically setting a 
unique cookie within each browser, the webapp (thanks to 
se s sion) maintains a browser-identifiable value of user 
for each browser. 

From the webapp’s perspective, it’s as if there are multiple 
values of user in the se s sion dictionary (keyed by 
cookie). From each browser’s perspective, it’s as if there is 
only ever one value of user (the one associated with their 
individual, unique cookie). 
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session @ work 


Managing Logins with Sessions 

Based on our work with qui ck_ses sion . py, we know we can sto re browser- 
specific state in session. No matter how many browsers interact with our 
webapp, each browser’s server-side data (a.k.a. state) is managed for us by Flask 
whenever session is used. 

Let 5 s use this new know-how to return to the problem of controlling web access to 
specific pages within the vsearch4web . py webapp. Recall that we want to get 
to the point where we can restrict who has access to the /viewlog URL. 

Rather than experimenting on our working vsearch4web . py code, let 5 s 
put that code to one side for now and work with some other code, which weil 
experiment with in order to work out what we need to do. Weil return to the 
vsearch4web . py code once weVe worked out the best way to approach this. 
We can then confidently amend the vsearch4web . py code to restrict access to 
/viewlog. 

Here 5 s the code to yet another Flask-based webapp. As before, take some time to 
read this code prior to our discussion of it. This is simple_webapp . py: 



Reaw Batte 



from flask import Flask 

app = Flask (_name ) 

@app.route ('/') 
def hello() -> str: 

return 'Helio from the simple webapp.' 

@app.route('/pagel') 
def pagel() -> str: 

return 'This is page 1.' 

@app.route('/page2') 
def page2() -> str: 

return 'This is page 2.' 

@app.route('/page3') 
def page3() -> str: 

return 'This is page 3.' 


if name 




main 


app.run(debug=True) 


This is simple_webc 

A"t "this s^fcage ih thi: 
book ; you should hav 

di-P-Pieul-fcy trcadihg i 

tode 3v\d uv)de\rs-tdy\c 
w ba*t “this webapp do 
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Ut $ Po logiw 

The simple_webapp . py code is straightforward: ali of the URLs are public in 
that they can be accessed by anyone using a browser. 

In addition to the default / URL (which results in the hello function executing), 
there are three other URLs, / 'pagel , / 'page2 , and /page3 (which invoke similarly 
named functions when accessed). All of the webapp ? s URLs return a specific 
message to the browser. 

As webapps go, this one is really just a shell, but will do for our purposes. We’d like 
to get to the point where / 'pagel , / 'page2 , and /page3 are only visible to logged- 
in users, but restricted to everyone else. We’re going to use FlasUs session 
technology to enable this functionality. 

Let 5 s begin by providing a really simple /login URL. For now, we’re not going to 
worry about providing an HTML form that asks for a login ID and password. All 
we’re going to do here is create some code that adjusts session to indicate that 
a successful login has occurred. 



Let's write the code for the /login URL below. In the space shown, provide 
code that adjusts session by setting a value for the logged_in key 
to True. Additionally, have the URL's function return the"You are now 
logged in" message to the waiting browser: 


/\dd 

kcw dode 


@app.route('/login') 



In addition to creating the code for the /login URL, you'll need to make two other changes to the code to 
enable sessions. Detail what you think these changes are here: 


o 

o 


you are here ► 


375 











login ready 


Wr / «7" 
i v Solution 


You were to write the code for the /login URL below. You were to provide 
code that adjusts session by setting a value for the logged_in key 
to True. Additionally, you were to have the URL's function return the "You 
are now logged in" message to the waiting browser: 


Set 'WdJ," key i, 

@app. route (' / login' ) w session dietionav-y -fco 

True . 

def do_login() -> str: 

sessionClo<^ed_jn*J — Tirue 
...•. Reiuirn this 

r*essac\e 

return You a r C r ov, Ioaeca m; the waitmg 

bvov/sev. 



In addition to creating the code for the /login URL, you needed to make two other changes to the code to 
enable sessions. You were to detail what you think these changes were: 


o 

o 


IA/e 


need *bo add 'session* *fco *tbe import line a*t *tbe *fcop o-f *tbe dode. 


We need *to se*t a value -fo\r *tbis y/ebapp*s sedv-et key. 


LcVs no*t 
.fov^et “to 
do these- 


Awewd the webapp'$ code to handle logiws 

We’re going to hold off on testing this new code until weVe added another 
two URLs: /logout and /status. Before you move on, make sure your copy of 
simple_webapp . py has been amended to include the changes shown below. 
Note: we’re not showing all of the webapp’s code here, just the new bits (which 
are highlighted): 


from flask import Flask, session 
app = Flask( name ) 



Rcmembev* 

*fco impovt 

U )) 

session . 


Add *tbe 

dodc -for *tbe 
Vloo^in* URL 


@app.route('/login') 
def do_login() -> str: 

session['logged_in'] = True 
return 'You are now logged in.' 


app.secret_key = 'YouWillNeverGuessMySecretKey' 

if _name_ == '_main_' : 

app.run(debug=True) 


£e-t a value *for -tbis 
webapp s sedv-et key 
(>whieb enables -tbe 
use of sessions). 
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Let's Po logout and Status Checkmg 

Adding the code for the /logout and /status URLs is our next task. 

When it comes to logging out, one strategy is to set the session dictionary 5 s 
logged_in key to False. Another strategy is to remove the logged_in key 
from session altogether. We’re going to go with the second option; the reason 
why will become ciear after we code the /status URL. 


Sharpen your pencil 




J \dd 

|o^OU“t 
dode heve. 


Let's write the code for the /logout URL, which needs to remove the 
logged_in key from the session dictionary, then return the"You are 
now logged out" message to the waiting browser. Add your code into the 
spaces below: 

Hin-t: i-f you've -folrgo-t-te» hov, •(» 

y«.ove a key a dicWavy, 

@app.route('/logout') typedir(dici)" ai the »> ' 

def do_logout () -> str: pironnp-t -Tov a list o-p avdilable 

difrtiohairy n, e -thocls. 


return 


With /logout written, we now turn our attention to /stotus, which returns one of two messages to the 
waiting web browser. 

The message"You are currently logged in" is returned when logged_in exists as a value in the 
session dictionary (and, by definition, is set to True). 

The message "You are NOT logged in" is returned when the session dictionary doesn't have a 
logged_in key. Note that we can 7 t check logged_in for False, as the /logout URL removes the key 
from the session dictionary as opposed to changing its value. (We haven't forgotten that we stili need 
to explain why we're doing things this way, and we'll get to the explanation in a while. For now, trust that 
this is the way you have to code this functionality.) 


Let's write the code for the /status URL in the space below: 
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logout status ready 


Wr / «7" 
i v Solution 


You were to write the code for the /logout URL, which needed to remove 
the logged_in key from the session dictionary, then return the "You 
are now logged out" message to the waiting browser: 


@app.route('/logout') 
def do_logout() -> str: 

sessior\.popOo^ed_mO 


t! Se P°P ""cthod to (remove 

!< ;?9 ed - ih " ke y ko» -the 
sessio» dictionary. 


return 


; You are now lo<)(jed ou*t/ 


With /logout written, you were to turn your attentiori to the /stotus URL, which returns one of two 
messages to the waiting web browser. 

The message "You are currently logged in" is returned when logged_in exists as a value in the 
session dictionary (and, by definition, is set to True). 


The message "You are NOT logged in" is returned when the session dictionary doesn't have a 
logged in key. 


You were to write the code for /stotus in the space below: 

@app.route('/status') 
def check_status() -> str: 

if l°99cd_jr/ in sessio^ 


t>oes -the "loggedjh" 

key exfct in the "session 
dictionary? 



return 


'Y<>u are durren*tly lo^ed in/ 



l£ yes, reiurn 
this messaae. 


return 


'Y<>u are NOT logted in/ 



|jf y\ 0 ) retur* 

•bWis wessa^c* 


Amend the webapp'$ code 
once more 

We’re stili holding off on testing this 
new version of the webapp, but here (on 
the right) is a highlighted version of the 
code you need to add to your copy of 
simple_webapp. py. 

Make sure you Ve amended your code to 
match ours before getting to the next Test 
Drive , which is coming up right after we 
make good on an earlier promise. 


@app.route('/logout') 
def do_logout() -> str: 

session.pop('logged_in') 
return 'You are now logged out. 


@app.route('/status') 
def check_status() -> str: 

if 'logged_in' in session: 

return 'You are currently logged in.' 
return 'You are NOT logged in.' 



Two new 

URL 

roules 


app.secret key = 'YouWillNeverGuessMySecretKey' 


if _name_ == '_main_ 

app.run(debug=True) 
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Why Not Check for False? 

When you coded the /login URL, you set the logged_in key to True in the session 
dictionary (which indicated that the browser was logged into the webapp). However, when 
you coded the /logout URL, the code didn’t set the value associated with the logged_ 
in key to False, as we preferred instead to remove all trace of the logged_in key 
from the session dictionary In the code that handled the /status URL, we checked 
the “login status” by determining whether or not the logged_in key existed in the 
session dictionary; we didn’t check whether logged_in is False (or True, for that 
matter). Which begs the question: why does the webapp not use False to indicate (i not logged 
in”? 

The answer is subtle, but important, and it has to do with the way dictionaries work in 
Python. To illustrate the issue, let 5 s experiment at the >>> prompt and simulate what can 
happen to the session dictionary when used by the webapp. Be sure to follow along 
with this session, and carefully read each of the annotations: 


Python 3.5.1 Shell 


»> 

»> session = dict() 

»> 

»> if session[' logged_in ']: 
print('Found it.') 


Traceback (most recent call last): 

File "<pyshell#47>", line 1, in <module> 
if session['logged_in']: 

KeyError: 'logged_in' 

»> 

»> if 'logged_in' in session: 
print('Found it.’) 


»> session [’ logged_in ' ] = True 
»> 

»> if 'logged_in' in session: 
print('Found it.’) 


Found it. 

»> 

»> if session [' logged_in' ] 
print(' Found it.’) 


Found it. 

»> I 


Ln: 115 Coi: 4 



_ Ctreaie a new, emp-ty 
did-tiomairy dalled "sessio*" 

JTVy to dhedk -fov* existe*£e ot a 

lo 3 ged_i« value usmg a* "i-f" stater»e*i 

Whoopsf The key doesn^t 

' exist yet> so we get a w k(eyfor-vov”, and 
ouv dode has dv-ashed as a vesult- 

Howevev, i-f we dhedk -fov existende using 
V, ouv dode doesn t dvash (theves no 
Yey£vrov") even though the key has no value- 

Let's assign a value to the w lo<^ed_m key. 

Chedking tov existende with w in W stili wo\rks, 
althou^h this time av*ound we get a positive 
vesult (as the key exists and has a value). 

Chedking with an it statement wovks too 
(now that the key has a value assodiated 
with it), ftoweveir, it the key is removed tvom 
the didtiona\ry (using the w poj>" method) this 
dode is onde again vulnevable to u kfeyErv*or" 


The above experimentation shows that it is not possible to check a dictionary for a key 5 s 
value until a key/value pairing exists. Trying to do so results in an KeyError. As it’s 
a good idea to avoid errors like this, the simple_webapp . py code checks for the 
existence of the logged_in key as proof that the browser’s logged in, as opposed to 
checking the key 5 s actual value, thus avoiding the possibility of a KeyError. 
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status login logout 



Let’s take the simple_webapp .py webapp for a spin to see how well th e/login, /logout, and 
/status URLs perform. As with the last TestDrive, we’re going to test this webapp using more than one 
browser in order to confirm that each browser maintains its own “login state” on the server. Let’s start 
the webapp from our operating systerrVs terminal: 

On Linux and Mac OS X: python3 simple_webapp.py 

On Windows: py -3 simple_webapp.py 

Let’s fire up Opera and check its initial login status by accessing the /status URL. As expected, the 
browser is not logged in: 



Attess ■the u /status' 

URL to detevWme 
whethev the bvowsev 
is logged m o\r *ot 


Q 127.0.0.1:5000/status 



+ 


© 127.0.0.1:5000/status 


□ 


You are NOT logged in. 



As yoi/ve ohly just 
stav-ted -the webapp, 

ahd this is youv -pivst 
i»te\radtioh with it, this 

r*essage doh-fiv-^s exaetly 
what you ; d expedi* that 
you a\re hot logged ih. 


Let’s simulate logging in, by accessing the /login URL. The message changes to confirm that the login 
was successful: 


fote. ssm$ /lo^m 
does e*attly 
is expedted o£ it 

The bvowsev is 
»ow logged 'mto 
the webapp- 

• • • 

r > 

Q 127.0.0.1:5000/login 

+ 

▼ 

< 

C :: : © 127.0.0.1 :5000/login 

¥ di 


You are now logged in. 
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Now that you are logged in, let’s confirm the status change by accessing the /status URL within Opera. 
Doing so confirms that the user of the Opera browser is logged in. If you use Chrome to check the 
status, too, you’ll see that the user of Chrome isn’t logged in, which is exactly what we want (as each 
user of the webapp—each browser—has its own state maintained by the webapp): 


Q 127.0.0.1:5000/status 


+ 


< 


© 127.0.0.1:5000/status 


You are currently logged in. 


The user o( 
the Opera 
browser is 
^urrehtly 
logged i*. 



0 127.0.0.1:5000/status 


\V_i 


C D 127.0.0.1:5000/status 


☆ 


You are NOT logged in. 


The user of the Chrome browser U the same 
dom ? uter) is *»ot* lo^ed m, as 7 our **ba ?? 
mamtams a se ? arate do ?7 of Chrome s sess.o» data. 

This is e*afctl 7 what we wa»t- 



To conclude, let’s access the /logout URL within Opera to teli the webapp that we are logging out of 
the session: 


Q 127.0.0.1:5000/logout 


+ 


< 


© 127.0.0.1:5000/logout 


You are now logged out. 


\ . . 

Retall that atdessm^ /logout 

removes the u lo<fted_i»" ke 7 trom 

this browser'S "sessio»", whidh has 

the effedt of simulati»^ a logout- 




Although we haven't asked any of our browser's 
users for a login ID or password, the /login, / 
logout , and /status URLs allow us to simulate 
what would happen to the webapp's session 
dictionary if we were to create the required HTML 
form, then hook up the fornis data to a backend 
“credentials” database. The details of how this 
might happen are very much application-specific, 
but the basic mechanism (i.e., manipulating 
session) is the same no matter what a specific 
webapp might want to do. 

Are we now ready to restrict access to the 
/pagel, /page2, and /page3 URLs? 


you are here ► 


381 





































ready to restrict? 


Caw We Now Restrict Access to URLs? 




Jim: Hey, Frank...what are you stuck on? 

Frank: I need to come up with a way to restrict access to the 
/pagel, /page2, and /page3 URLs... 

Joe: It can’t be that hard, can it? You Ve already got the code 
you need in the function that handles / status ... 

Frank: ...and it knows if a user’s browser is logged in or not, 
right? 

Joe: Yeah, it does. So, all you have to do is copy and paste 
that checking code from the function that handles / status 
into each of the URLs you want to restrict, and then you’re 
home and dry! 

Jim: Oh, man! Copy and paste...the web developers Achilles’ 
heel. You really don’t want to copy and paste code like that... 
it can only lead to problems down the road. 

Frank: Of course! GS 101... Fll create a function with the 
code from / status , then call that function as needed within the 
functions that handle the /pagel , / 'page2 , and /page3 URLs. 
Problem solved. 

Joe: I like that idea...and I think it’ll work. (I knew there was 
a reason we sat through all those boring GS lectures.) 

Jim: Hang on...not so fast. What you 5 re suggesting with a 
function is much better than your copy-and-paste idea, but 
Fm stili not convinced it 5 s the best way to go here. 

Frank and Joe (together, and incredulously): Whatfs not to 
like?!?!? 

Jim: It bugs me that you’re planning to add code to the 
functions that handle the /pagel , / 'page2 , and /page3 URLs 
that has nothing to do with what those functions actually 
do. Granted, you need to check whether a user is logged in 
before granting access, but adding a function call to do this 
to every URL doesn’t sit quite right with me... 

Frank: So what’s your big idea, then? 

Jim: If it were me, Fd create, then use, a decorator. 

Joe: Of course! That’s an even better idea. Let’s do that. 
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Copy-awd-Paste Is Rarely a (rood Idea 

Let 5 s convince ourselves that the ideas suggested on the last page are not the best 
way to approach the problem at hand—namely, how best to restrict access to 
specific web pages. 

The first suggestion was to copy and paste some of the code from the function 
that handles the /status URL (namely, the check_status function). Here 5 s the 
code in question: 


This is the 
dode to £opy 
a«d paste- 



@app. route (' / status ') 
def check_status() -> str: 

if 'logged_in' in session: 

return 'You are currently logged in.' 
return 'You are NOT logged in.' 




This dode vetuvns S 
di-fteneh-t message based 
on whe-ther on not the 
wsen's bnowsen is logged in 


Here’s what the pagel function currently looks like: 


@app.route('/pagel') 
def pagel() -> str: 

return 'This is page 1.' 


This is the page- 
speei-f ie -Pundtionality. 


If we copy and paste the highlighted code from check_status into pagel, 
the latter’s code would end up looking like this: 


@app.route('/pagel') 
def pagel() -> str: 

if 'logged_in' in session: 

return 'This is page 1.' 
return 'You are NOT logged in.' 



Cbedk it tbe usens bvowsev 
is lo^ed ih -* 

- - then do the paje—spedi-Pid -fundtionality- 


_ 

The above code works, but if you were to repeat this copy-and-paste activity 
for the /page2 and /page3 URLs (as well as any other URLs you were to add to 
your webapp), you’d quickly create a maintenance nightmare , especially when you 
consider all the edits you’d have to make should you decide to change how your 
login-checking code works (by, maybe, checking a submitted user ID and password 
against data stored in a database). 


0the\rwise, iKvPoirrrt the usev* 
that they ave Kot lo^ed in. 


Put shared code into its own function 

When you have code that you need to use in many different places, the classic 
solution to the maintenance problem inherent in any copy-and-paste “quick fix” is 
to put the shared code into a function, which is then invoked as needed. 

As such a strategy solves the maintenance problem (as the shared code exists in 
only one place as opposed to being copied and pasted willy-nilly), let’s see what 
creating a login-checking function does for our webapp. 
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Creating a Function Helps, Put... 


Let’s create a new function called check_logged_in, which, when invoked, 
returns True if the user 5 s browser is currently logged in, and False otherwise. 


It 5 s not a bigjob (most of the code is already in check_status); here 5 s how 
we’d write this new function: 


def check_logged_in() -> bool: 
if 'logged_in' in session: 

return True 
return False 


Rathev ihah v-eWning a message, 

«-wis tode ve-fcuvus a booleah based oh 

wbetheir ov hot the use*-'s bvowsev is 
logged ih. 


With this function written, let’s use it in the pagel function instead of that copied 
and pasted code: 


lAfeVe ehedkmg 
i-f >we ave t* 
logged i*- 



@app.route('/pagel') 
def pagel() -> str: 

if not check_logged_in() 

return 'You are NOT logged in. 
return 'This is page 1.' 





This strategy is a bit better than copy-and-paste, as you can now change how the 
login process works by making changes to the check_logged in function. 
However, to use the check_logged_in function you stili have to make similar 
changes to the page2 and page3 functions (as well as to any new URLs you 
create), and you do that by copying and pasting this new code from pagel into 
the other functions... In fact, if you compare what you did to the pagel function 
on this page with what you did to page 1 on the last page, it’s roughly the same 
amount of work, and it’s stili copy-and-paste! Additionally, with both of these 
“Solutions,” the added code is obscuring what pagel actually does. 


Call -the £he£k_Jogged_jh” -Puh^-fcioh -to 

detevmme the logi* status, the* ati aitord i 


This todt only evev vuns 
i-f the usev s bvowsev is 
logged in. 


It would be nice if you could somehow check if the user’s browser is logged in 
without having to amend any of your existing function’s code (so as not to obscure 
anything). That way, the code in each of your webapp’s functions can remain 
directly related to what each function does, and the login status-checking code 
won’t get in the way. If only there was a way to do this? 


As we learned from our three friendly developers—Frank, Joe, andjim—a few 
pages back, Python includes a language feature that can help here, and it goes by 
the name decorator. A decorator allows you to augment an existing function 
with extra code, and it does this by letting you change the behavior of the existing 
function without having to change its code. 


If you’re reading that last sentence and saying: (i What?!?!?’\ don’t worry: it does 
sound strange the first time you hear it. After ali, how can you possibly change 
how a function works without changing the function’s code? Does it even make 
sense to try? 


Let’s find out by learning about decorators. 
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YouVe Peew Using Pecorators AII Along 

YouVe been using decorators for as long as youVe written webapps with Flask, 
which you started back in Chapter 5. 

Here 5 s the earliest version of the hello_f lask. py webapp from that chapter, 
which highlights the use of a decorator called @app . route, which comes with 
Flask. The @app . route decorator is applied to an existing function (hello in 
this code), and the decorator augments the function it precedes by arranging to 
call hello whenever the webapp processes the / URL. Decorators are easy to 
spot; they’re prefixed with the @ Symbol: 



Note how, as a user of the @app . route decorator, you have no idea how the 
decorator works its magic. All you’re concerned with is that the decorator does 
what it promises: links a given URL with a function. All of the nitty-gritty, behind- 
the-scenes details of how the decorator works are hidden from you. 

When you decide to create a decorator, you need to peek under the covers and 
(much like when you created a context manager in the last chapter) hook into 
Python’s decorator machinery. There are four things that you need to know and 
understand to write a decorator: 


o 

o 

o 

o 


How to create a function 

How to pass a function as an argument to a function 

How to return a function from a function 

How to process any number and type of function arguments 


You’ve been successfully creating and using your own functions since Chapter 
4, which means this list of “four things to know” is really only three. Let’s take 
some time to work through items 2 through 4 from this list as we progress toward 
writing a decorator of our own. 
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Pass a Function to a Function 

It’s been a while, but way back in Chapter 2 we introduced the notion that 
everything is an object in Python. Although it may sound counterintuitive, the 
“everything” includes functions, which means functions are objects, too. 

Clearly, when you invoke a function, it runs. However, like everything else in 
Python, functions are objects, and have an object ID: think of functions as 
“function objects.” 

Take a quick look at the short IDLE session below. A string is assigned to a 
variable called msg, and then its object ID is reported through a call to the id 
built-in function (BIF). A small function, called hello, is then defined. The 
hello function is then passed to the id BIF that reports the function’s object ID. 
The type BIF then confirms that msg is a string and hello is a function, and 
finally hello is invoked and prints the current value of msg on screen: 


□ 

Pass a function to a 
function. 

□ 

Return a function 
from a function. 

□ 

Process any number/ 
type of arguments. 


/ 

We II eheek otf eaeh 
ton ' ple-fced topid as we 
Work through this «aterial. 


The "id" W «?«*** the 

u,, r e objefct idertitier tor 

ar»y objett provided to 't- 



Python 3.5.1 Shell 


»> 

»> msg = "Hello from Head First Python 2e" 
»> id (msg) 

4385961264 
»> def hello () : 

print (msg) 


»> id (hello) 

4389417984 
»> type (msg) 

<class 'str'> 

»> type (hello) 

<class 'function 
»> hello () 

Hello from Head First Python 2e 
»> 



The "type" BIF 
reports oh ah 
objedts type. 


Ln: 20 Coi: 4 


We were a little devious in not drawing your attention to this before we had you 
look at the above IDFE session, but...did you notice how we passed hello to the 
id and type BIFs? We dkhft invoke hello; we passed its name to each of the 
functions as an argument. In doing so, we passed a function to a function. 


Functions can take a f unction as an argument 

The calls to id and type above demonstrate that some of Python’s built-in 
functions accept a function as an argument (or to be more precise: a function object). 
What a function does with the argument is up to the function. Neither id nor 
type invokes the function, although it could have. Fet 5 s see how that works. 
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Ihvokiwg a Passed Function 

When a function object is passed as an argument to a function, the receiving 
function can invoke the passed-in function object. 

Here’s a small function (called apply) that takes two arguments: a function object 
and a value. The apply function invokes the function object and passes the value 
to the invoked function as an argument, returning the results of invoking the 
function on the value to the calling code: 


□ 

Pass a function to a 
function. 

□ 

Return a function 
from a function. 

□ 

Process any number/ 
type of arguments. 


The "apply" 

WW attepts 

3 -function objecT 
as a* argument 
The “objcfrt" — 

anno-taton helps 
{p eon-firm our 

intentio* here 
(and the use o-f 
the argument 
name “*fun£* 
is a dommon 
eonvention). 


func.py - /Users/paul/Documents/func.py (3.5.1) 




def apply (fune: object, value: 
return fune(value) 

/N 


object) -> object: 


Ln: 6 Coi: 0 


type) ean be passed 
as the seeond 
argument Again, the 
annotations hint at 
whats allowed as an 
argument type here* 
any objeet 


The tundtion (passed as an argument) is invoked, with the “value” passed 
to it as its only argument The resuit ot this tundtion dall is returned 

trom the “apply” tundtion. 


Note how apply’s annotations hint that it accepts any function object together 
with any value, then returns anything (which is all very generic). A quick test of 
apply at the >>> prompt confirms that apply works as expected: 


The “apply” 

-funetion runs a 

blAKth ot BlFs —> 

against some 
values (and works 
as c*peeted)- 


Python 3.5.1 Shell 


»> 

»> apply (print, 42) 
42 

»> apply (id, 42) 
4297539264 
»> apply (type, 42) 
<class 'int’> 

»> apply (len, 

6 

»> apply (type, 
<class 'functioni 
»> ! 


The apply" tu neti Oh 
takes any object tor 
value . /n this example, 

't takes itself as “value* 
and don-Pirms that it s a 
tunetion. 


Ln: 110 Coi: 4 


Marvin' ) 
apply) <r 


In eaeh o-f these 
cxamplthe 
>- |t— ( tivst a\rgur»eh-t io 
'apply" is assianed 
l to the "-fuhC" 

a\rgu»«eiit (above). 


If you’re reading this page and wondering when you’d ever need to do something 
like this, don’t fret: well get to that when we write our decorator. For now, 
concentrate on understanding that it’s possible to pass a function object to a 
function, which the latter can then invoke. 
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Functions Can Pe Nested Inside Functions 

Usually, when you create a function, you take some existing code and make it 
reusable by giving it a name, and using the existing code as the function’s suite. 
This is the most common function use case. However, what sometimes comes as a 
surprise is that, in Python, the code in a function 5 s suite can be any code, including 
code that defines another function (often referred to as a nested or inner function). 
Even more surprising is that the nested function can be returned from the outer, 
enclosing function; in effect, what gets returned is a function object. Let 5 s look at a 
few examples that demonstrate these other, less common function use cases. 

First up is an example that shows a function (called inner) nested inside another 
function (called outer). It is not possible to invoke inner from anywhere other 
than within outer 5 s suite, as inner is local in scope to outer: 


0 

Pass a function to a 
function. 

□ 

Return a function 
from a function. 

□ 

Process any number/ 
type of arguments. 


The 'Weir 

-funfi-tion is 
de-fmed wrthih “the 
CKdIosmJ s 

suite- 


def outer(): 
def inner 
print( 


0 : 

' This 


is inner.') 


print('This is outer, invoking inner.' 
inner() 

The "ihhev-" tWtio* 



is ihVoked -fv-om 

w / i) 

outeir . 


) 


When outer is invoked, it runs all the code in its suite: inner is defined, the call 
to the print BIF in outer is exeeuted, and then the inner function is invoked 
(which calls the print BIF within inner). Here’s what appears on screen: 

The prmted 

**essa$es appeav m 
the ovdev : outev" ^ 
-fivst, then W ’mnev- - 

Fooking at this simple example, you might find it hard to think of a situation 
where creating a function inside another function would be useful. However, 
when a function is complex and contains many lines of code, abstracting some of 
the function’s code into a nested function often makes sense (and can make the 
enclosing function’s code easier to read). 

A more common usage of this technique arranges for the enclosing function to 
return the nested function as its value, using the return statement. This is what 
allows you to create a decorator. 

So, let’s see what happens when we return a function from a function. 


This is outer, invoking inner 
This is inner. 


} 


When would you ever use this? 
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Return a Function frow a Function 

Our second example is very similar to the first, but for the fact that the outer 
function no longer invokes inner, but instead returns it. Take a look at the code: 


The 'Weir 
-functio* is stili 
de*C’med Wrthi* 
w ou*tev • 


def outer(): 

def inner(): 

print('This is inner.') 



print('This is outer, returning inner.') 
return inner 


0 

Pass a function to a 
function. 

□ 

Return a function 
from a function. 

□ 

Process any number/ 
type of arguments. 


^- The . «Wementdoesnot 

"Tr : VnsU * d > * «W 

the .hhor Wiio« objeci to the 


Let’s see what this new version of the outer function does, by returning to the 
IDLE shell and taking outer for a spin. 


Note how we assign the resuit of invoking outer to a variable, called i in this 
example. We then use i as if it were a function object—first checking its type 
by invoking the type BIF, then invoking i as we would any other function (by 
appending parentheses). When we invoke i, the inner function executes. In 
effect, i is now an alias for the inner function as created inside outer: 


The "ou-teC -fuwi-tio» 
is iiwoked. 


The vesult 
tallis w ou*tev ) 
is assi^ed 
{p a vaviable 


called 






»> 



Python 3.5.1 Shell 


»> i = outer () 

This is outer, returning inner. 
»> type (i) 

Cclass ' function'> 

»> i 0 


We chedk that "i" i s , 

i« -fact, a -PuhC-tioh. 



We mvoke V a*d—voilal-the 
Wv" WWs tode e^etutes. 


So far, so good. You can now return a function from a function, as well as send a 
function to a function. You’re nearly ready to put all this together in your quest to 
create a decorator. There’sjust one more thing you need to understand: creating a 
function that can handle any number and type of arguments. Let’s look at how to 
do this now. 
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argument lists 


Acceptmg a List of Arguments 


Imagine you have a requirement to create a function (which we’ll call myf unc in 
this example) that can be called with any number of arguments. For example, you 
might call myf unc like this: 



myfunc(10) 


One 


or you might call myfunc like this: 

c 

myfunc() 


No av-^umCh*ts 


0 

Pass a function to a 
function. 

0 

Return a function 
from a function. 

□ 

Process any number/ 
type of arguments. 

k 

Y°uVe neavly theve. 
One more topid to 


toner, and the« 


youll be veady to 
treaie a detoralor. 


or you might call myfunc like this: 

myfunc(10, 20, 30, 40, 50, 60, 70) 


In fact, you might call myfunc with any number of arguments, with the proviso 
that you don’t know ahead of time how many arguments are going to be provided. 


Uidh, 

ln ^" s exa**>ple, are all 
Ku«,be«, but dould be 
anythmg: n^mbers, stv-ihfls, 
booleans, (is-t. 3 


As it isn’t possible to define three distinet versions of myfunc to handle each of 
the three above invocations, the question becomes: is it possible to accept any number 
of arguments in a function? 


Use * to accept an arbitrary list of arguments 


Python provides a special notation that allows you to specify that a function can 
take any number of arguments (where “any number” means “zero or more”). 
This notation uses the * character to represent any number , and is combined with 
an argument name (by convention, args is used) to specify that a function can 
accept an arbitrary list of arguments (even though *args is technically a tuple). 

Here’s a version of myfunc that uses this notation to accept any number of 
arguments when invoked. If any arguments are provided, myfunc prints their 
values to the screen: 


Tliink of * 
as nteaning 
"expand to a 
list of values.” 


Tbe "*avy 
ttcrtdtiOK medus 
u zjCV“o ov roove 



myfunc.pyY /Users/paul/Desktop/NewBook/ch10/myfunc.py (3.5.1) 


def myfunc (*args): 
for a in args: 

print(a, end=' ' ) 
if args: 

print () 




TUk ot avgs" as a list 
f a < r 9<«»e>’>ts, whidh da* 
be prodessed like a*y 

eir list (even though 
't s a tuple). 3 

Ln: 7 Coi: 0 


Arv-a^ges to display tbe list ot a*-gume»t 
values oy\ a smgle line 
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function decorators 


Processing a List of Arguments 

Now that myf unc exists, let 5 s see if it can handle the example invocations from 
the last page, namely: 


myfunc(10) 
myfunc() 

myfunc(10, 20, 30, 40, 50, 60, 70) 


0 

Pass a function to a 
function. 

0 

Return a function 
from a function. 

□ 

Process any number/ 
type of arguments. 


Here 5 s another IDLE session that confirms that myfunc is up to the task. No 
matter how many arguments we supply (including none), myfunc processes them 
accordingly: 


Ko r*attev 
-the nurttbev" ot 
avtyAwcnts ^rovided, 
w r*ydoes 
-the v~i$ht thm^ 

(i.C., fVOdCSSCS 

its av^uwcwtsj 

mattev- how ^ 3 ^ 7 )- 


Python 3.5.1 Shell 


»> 

»> myfunc (10) 
10 

»> myfunc () 


^ M ^ur^en-b, 
'r*y+Uhd” does hothihg. 


»> myfunc (10, 

20, 30, 40, 50, 60, 

70) 

10 20 30 40 50 

60 70 


»> 



»> myfunc (1, 

two' , 3, 'four' , 5, 

'six' , 7) 

1 two 3 four 5 

six 7 


>>> 



\ \ Ln: 37 Coi: 4 



You da* CVCn r*i% ay\d "*atdh “the 
-t V p C s the values providcd, and 

«jyW" sti» does the vi^t thi^- 


works oh the way in, too 


If you provide a list to myfunc as an argument, the list (despite potentially 
containing many values) is treated as one item (i.e., it’s one list). To instruet the 
interpreter to expand the list to behave as if each of the list’s items were an 
individual argument, prefix the list’s name with the * character when invoking the 
function. 

Another short IDLE session demonstrates the difference using * can have: 

The list is 
pv-odessed as a 
sin^le avehent 
to the -fundtion- 




Python 3.5.1 Shell 


»> 

»> values = [1, 2, 3, 5, 7, 11] 
»> 

»> myfunc (values) 

[1, 2, 3, 5, 7, 11] 

»> 

»> myfunc (*values) 

1 2 3 5 7 11 
»> 

»> I 


A list ot 
si* integevs 


IVheh a list is piretixed with 

>± expahds to a list of individual 
av-guments. 


Ln: 15 Coi: 4 
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argument dictionaries 


Acceptiwg a Pictiowary of Arguwewts 

When it comes to sending values into functions, it’s also possible to provide the 
names of the arguments together with their associated values, then rely on the 
interpreter to match things up accordingly. 

You first saw this technique in Ghapter 4 with the search41etters function, 
which—you may recall—expects two argument values, one for phrase and 
another for letters. When keyword arguments are used, the order in which the 
arguments are provided to the search41etters function doesn’t matter: 


This is ov\t 
way to 
mvoke the 

-fundtion- 


0 

Pass a function to a 
function. 

0 

Return a function 
from a function. 

□ 

Process any number/ 
type of arguments. 



search4letters(letters= 1 xyz 1 , phrase= 1 galaxy') 



This is 
«toother 
way to 

ihvokc a 
tuhdtioh. 



def search41etters(phrase:str, letters:str= 1 aeiou 1 ) -> set: 


Like with lists, it’s also possible arrange for a function to accept an arbitrary 
number of keyword arguments—that is, keys with values assigned to them (as with 
phrase and letters in the above example). 


Use ** to accept arbitrary keyword arguwewts 

In addition to the * notation, Python also provides * *, which expands to a 
collection of keyword arguments. Where * uses args as its variable name (by 
convention), ** uses kwargs, which is short for “keyword arguments.’ 5 (Note: 
you can use names other than args and kwargs within this context, but very 
few Python programmers do.) 

Let’s look at another function, called myfunc2, which accepts any number of 
keyword arguments: 


Tkink ol ** as 
meaningf "expand 
to a dictionary ol 
keys and values.” 


myfunc.py - /Users/paul/Desktop/NewBook/ch10/myfunc.py (3.5.1) 


VVrthin tunfction» 
"kwav-y bebaves 
ms-t like any 
diiCicmavy 


def myfunc (*args): 
for a in args: 

print(a, end=’ ') 
if args: 

print () 

def myfunc2 (**kwargs): 
for k, v in kwargs 
print (k, v, sep 
if kwargs: 
print () 


The w **" -tdls 
“the -Puhdtioh io 
exped-t keyword 
argumehts. 


items () : 1 

= '' , end= 1 ' ) 



Ln: 1 3 Coi: 0 


l^ke eadh key 
ahd value pairihg 
the didtiohary, 
3hd display it oh 
sdveeh. 
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Processing a Pictionary of Arguwewts 

The code within myfunc2 5 s suite takes the dictionary of arguments and processes 
them, displaying all the key/value pairings on a single line. 

Here’s another IDLE session that demonstrates myfunc2 in action. No matter 
how many key/value pairings are provided (including none), myfunc2 does the 
right thing: 


0 

Pass a function to a 
function. 

0 

Return a function 
from a function. 

□ 

Process any number/ 
type of arguments. 


Two 

keywovd 
av^urweirrts 
yvovided 


# C ® _ Python 3.5.1 Shell 

»> ^ ^ 
p» myfunc2(a=10, b=2 

b->20 a->10 

»> myfunc2 () ^- 

»> myfunc2 (a=10 , b=2 

b->20 f->60 d->40 c-> 

»> / 
»> 

■ 0 ) Pv*ovidih 0 ko av^umen*^ 

— ish t issuc. 

!0, c=30 f d=40, e=50, f=60) 

►30 e->50 a->10 

v 


Ln: 24 Coi: 4 


Vow da* pvovide a*y nwbev- ot key^ovd 
arenis, a«d does the tHm* 


** works om the way in, too 


You probably guessed this was coming, didn’t you? As with *args, when you 
use **kwargs it 5 s also possible to use ** when invoking the myfunc2 function. 
Rather than demonstrate how this works with myfunc2, we’re going to remind 
you of a prior usage of this technique from earlier in this book. Back in Ghapter 
7, when you learned how to use Python’s DB-API, you defined a dictionary of 
connection characteristics as follows: 


/\ diC-tiofidv-y o-f 
key/value yaivmjs 


dbconfig = 



'host': '127.0.0.1', 

'user': 'vsearch', 

'password': 'vsearchpasswd ', 

'database': 'vsearchlogDB', } 


When it came time to establish a connection to your waiting MySQL (or 
MariaDB) database server, you used the dbconfig dictionary as follows. Notice 
anything about the way the dbconfig argument is specified? 


Does -this look 

-Pamiliar? 



conn = mysql.connector.connect(**dbconfig) 


By prefixing the dbconfig argument with **, we teli the interpreter to treat the 
single dictionary as a collection of keys and their associated values. In effect, it’s as 
if you invoked connect with four individual keyword arguments, like this: 

conn = mysql.connector.connect('host' = '127.0.0.1', 'user'='vsearch', 

'password'='vsearchpasswd', 'database'='vsearchlogDB') 
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arg ument anything 


Acceptmg Any Number and Type of 
Function Arguments 

When creating your own functions, it 5 s neat that Python lets you accept a list of 
arguments (using *), in addition to any number of keyword arguments (using **). 
What 5 s even neater is that you can combine the two techniques, which lets you 
create a function that can accept any number and type of arguments. 

Here’s a third version of myf unc (which goes by the shockingly imaginati ve 
name of myfunc3). This function accepts any list of arguments, any number of 
keyword arguments, or a combination of both: 


0 

Pass a function to a 
function. 

0 

Return a function 
from a function. 

□ 

Process any number/ 
type of arguments. 


The oviamal 

« c )i 

y/ovks 

wi-fch a*y list - 

avgumebts. 

The 

-Pubdtiob y/ovks -^ 

y/ith a^y amoubt 
of key/value paivs. 

The 

fubfctiob y/ovks y/ith 
a*y mputi y/hethev 3 
lis-t of av^uw^ts, a 
bunth of key/value 
paivs, ov both- 



myfunc.py - /Users/paul/Desktop/NewBook/ch10/myfunc.py (3.5.1) 


def myfunc (*args): 
for a in args: 

print(a, end=' ') 
if args: 

print () 

def myfunc2 (**kwargs): 

for k, v in kwargs.items(): 

print (k, v, sep='->', end=' ') 
if kwargs: 
print () 

def myfunc3 (*args, **kwargs): 
if args: 

for a in args: 

print (a, end=' ') 
print () 
if kwargs: 

for k, v in kwargs.items(): 

print (k, v, sep=’->', end=' ') 
print () 


Ln: 7 Coi: 0 



Both 

u *avgs" a*d 

W **kwa*y 

appeav ob 
the u def w 
libe- 


This short IDLE session showcases myfunc3: 


Wovks 
y/ith bo 
av^umebts 



Wovks y/ith a 
dorubibatiob 
of a list 
abd keywovd 
avgur^ebts 


Python 3.5.1 Shell 


Wovks y/ith a list 


»> 

»> myfunc3() 

»> myfunc3(l, 2, 3) 

12 3 

»> myfunc3 (a=10 , b=20, c=30) - 

a->10 b->20 c->30 

»> myfunc3(l, 2 , 3 , a=10, b=20, c=30) , 

12 3 av^umCbts 

a->10 b->20 c->30 

»> | 

Ln: 68 Coi: 4 


Wovks y/ith 
keywovd 
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function decorators 


A Recipe for Creating a Function Pecorator 

With three items marked in the checklist on the right, you now have an 
understanding of the Python language features that allow you to create a 
decorator. All you need to know now is how you take these features and combine 
them to create the decorator you need. 

Just like when you created your own context manager (in the last chapter), creating 
a decorator conforms to a set of rules or recipe. Recall that a decorator allows you 
to augment an existing function with extra code, without requiring you to change 
the existing function 5 s code (which, we’ll admit, stili sounds freaky). 

To create a function decorator, you need to know that: 


0 

Pass a function to a 
function. 

0 

Return a function 
from a function. 

0 

Process any number/ 
type of arguments. 

V 

1/VcVc ready -to 
have a go at 


writmg ou\r owk 
decorator. 



A decorator is a function 

In fact, as far as the interpreter is concerned, your decorator is just another 
function , albeit one that manipulates an existing function. Let’s refer to this 
existing function as the decoratedfunction from here on in. Having made it 
this far in this book, you know that creating a function is easy: use Python’s 
def keyword. 



A decorator takes the decorated function as an argument 

A decorator needs to accept the decorated function as an argument. To 
do this, you simply pass the decorated function as a function object to your 
decorator. Now that you Ve worked through the last 10 pages, you know 
that this too is easy: you arrive at a function object by referring to the 
function without parentheses (i.e., usingjust the function’s name). 



A decorator returns a new function 

A decorator returns a new function as its return value. Much like when 
outer returned inner (a few pages back), your decorator is going to do 
something similar, except that the function it returns needs to invoke the 
decorated function. Doing this is— dare we say it? —easy but for one small 
complication, which is what Step 4 is all about. 



A decorator maintains the decorated functions signature 

A decorator needs to ensure that the function it returns takes the same 
number and type of arguments as expected by the decorated function. 
The number and type of any function’s arguments is known as its 
signature (as each function’s def line is unique). 


It’s time to grab a pencil and put this information to work creating your first decorator. 
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what’s the motivation? 


Recap: We Need to Restrict Access to 
Certam URls 



We re trying to 
avo id copying and 
pasting all that login- 
status-checking code. 


WeVe been working with the simple_webapp . py code, and we need our 
decorator to check to see whether the user’s browser is logged in or not. If it 
is logged in, restricted web pages are visible. If the browser isn’t logged in, the 
webapp should advise the user to log in prior to viewing any restricted pages. 

We’ll create a decorator to handle this logic. Recall the check_status function, 
which demonstrates the logic we want our decorator to mimic: 


VVc warrt “to avoid 
dopY m 9 

yastmg this dode- 



@app. route (' / status ') 
def check_status() -> str: 

if 'logged_in' in session: 

return 'You are currently logged in.' 
return 'You are NOT logged in.' 




Rcmcmbcv: this to&t 
rctulrhS £ 

>*essage based ok whether 
0>r the usev-^s bvowsev 
,s ^°99cd ih. 
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Function Pecorator 

1 in our list, you had to create a new function. Remember: 

A decorator is a function 

In fact, as far as the interpreter is concerned, your decorator is just another 
function , albeit one that manipulates an existing function. Let’s refer to this 
existing function as the decoratedfunction from here on in. You know that 
creating a function is easy: use Python’s de f keyword. 

Complying with item 2 involves ensuring your decorator accepts a function object 
as an argument. Again, remember: 


Your decorator needs to accept the decorated function as an argument. 

To do this, you simply pass the decorated function as a function object to 
your decorator. You arrive at a function object by referring to the function 
without parentheses (i.e., using the function’s name). 


o A decorator takes the decorated function as an argument 


Creating a 


To comply with item 


o 


V, r 


■ j w m ^ wp ■ ■ ■ 


Let's put your decorator in its own module (so that you can more easily 
reuse it). Begin by creating a new file called checker . py in your text 
editor. 


Put tke decorators 
"det” Ime heve- 




You're going to create a new decorator in checker . py called check_ 
logged_in. In the space below, provide your decoratoks def line. Hint: 
use fune as the name of your function object argument: 


tliere .are no 

Dumb Questipns 



Does it matter where on my system I create checker. py? 


Yes. Our plan is to import checker. py into webapps that need it, so you need to ensure that the interpreter can find it when your 
code includes the import checker line. For now, put checker. py in the same folder as simple webapp. py. 
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decorator takes shape 


/ cV" 

i v Solution 


We decided to put your decorator in its own module (so that you can more 
easily reuse it). 

You began by creating a new file called checker . py in your text editor. 

Your new decorator (in checker . py) is called check_logged_in 
and, in the space below, you were to provide your decoratoks def line: 


de-P dhedk_Jo<^ed__m (•fund): 

/K The "dhedkjo^edj»" dedoraW -takes 
V— a sin^le av^wmext the •futtdtion objett ot 
the dedov-aied -fwndtion- 


That'$ almost too easy, i$n't it? 

Remember: a decorator is just anotherfunction , which takes a function object as an 
argument (fune in the above def line). 

Let’s move on to the next item in our “create a decorator” recipe, which is a little 
more involved (but not by much). Recall what you need your decorator to do: 



A decorator returns a new function 

Your decorator returns a new function as its return value. Just like when 
outer returned inner (a few pages back), your decorator is going to do 
something similar, except that the function it returns needs to invoke the 
decorated function. 


Earlier in this chapter, you met the outer function, which, when invoked, 
returned the inner function. Here’s outer’s code once more: 


^11 o£ this fcode^ 
is m the w ouW’ 
•fuKfrtions suite- 



def outer () : 

def inner () : 

print('This is inner.') 

* j 

print('This is outer, returning inner.') 
return inner 

/T\ 


The w irmev” -Punetion is 
»«ted inside u outev w . 


The w mncv w -Pundtion obje^t is vetuvned as the 
vesult o-P invoking w outcv w . Note the ladk o-P 
paventheses a-Ptev u ‘mnev- w , as weVe vetuvnin^ a 
tundtion objedt- We ave ^not* invokin^ w irmev . 
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function decorators 



Now that you've written your decorator's def line, let's add some code to 
its suite. You need to do four things here. 


1. Define a nested function called wrapper that is returned by check_ 
logged_in. (You could use any function name here, but, as you'll see in 
a bit, wrapper is a pretty good choice.) 


2. Within wrapper, add some of the code from your existing check_ 
status function that implements one of two behaviors based on 
whether the user's browser is logged in or not.To save you the page-flip, 
here's the check_status code once more (with the important bits 
highlighted): 

@app. route (' / status ') 
def check_status() -> str: 

if 'logged_in' in session: 

return 'You are currently logged in.' 
return 'You are NOT logged in.' 


3. As per item 3 of our decorator-creating recipe, you need to adjust 
the nested functioni code so that it invokes the decorated function (as 
opposed to returning the "You are currently logged in" message). 

4. With the nested function written, you need to return its function object 
from check_logged_in. 

Add the required code to check_logged_in , s suite in the spaces 
provided below: 


def check_logged_in(fune): 



Doni ( orge-fc 
i» \reiu\rn -the 
nested -(W-tioh. 



2. a»d 3. 

Add 


Code you 
Wnt -the 

nested 

TUhe^iort to 
exedu te- 
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decorator almost there 


/ cV" 

i v Solution 


With your decorator's def line written, you were to add some code to its 
suite. You needed to do four things: 


1. Define a nested function called wrapper that is returned by check 
logged in. 


2. Within wrapper, add some of the code from your existing check 
status function that implements one of two behaviors based on 
whether the user's browser is logged in or not. 


3. As per item 3 of our decorator-creating recipe, adjust the nested 
function's code so that it invokes the decorated function (as opposed to 
returning the "You are currently logged in" message). 


4. With the nested function written, return its function object from 

check logged in. 


You were to add the required code to check_logged_in , s suite in the 
spaces provided: 


J\ *es*ted 

u de£ w li*e 
s-tavb 
Wappev 
-Cunfrtio*. 



Did you ircmcrhbcv 
to re-fcuirh -the 
»es-fced W-fcioh? 


def check_logged_in(func): 

det \wvapf>evO: l( u se/s 

.f n , . , . , _■—' birowseir is logged ih... 

i-r m sessio*: « 

. —. ...mvoke *tbe 

dedov-ated WW 

\rctu\r* 'You av-e NOT logged i*- ; ^ 

\re*tu\r* y/\rappev- usc,r s 

. birowseir is*'-t 

logged i*, v-etuirr 

appiropHate r*ess 



Can you see wby the nested function is called "wrapper"? 

If you take a moment to study the decorator’s code (so far), you’11 see that the nested 
function not only invokes the decorated function (stored in fune), but also augments 
it by wrapping extra code around the call. In this case, the extra code is checking to see 
if the logged_in key exists within your webapp’s se s sion. Critically, if the user’s 
browser is not logged in, the decorated function is never invoked by wrapper. 
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The Final Step: Handling Arguments 

We are nearly there—the “guts” of the decorator 5 s code is in place. What remains 
is to ensure the decorator handles the decorated function’s arguments properly, no 
matter what they might be. Recall item 4 from the recipe: 

Your decorator needs to ensure that the function it returas takes the same 
number and type of arguments as expected by the decorated function. 


o A decorator maintains the decorated functions signature 


When a decorator is applied to an existing function, any calls to the existing 
function are replaced by calls to the function returned by the decorator. As you 
saw in the solution on the previous page, to comply with item 3 of our decorator- 
creation recipe, we retura a wrapped version of the existing function, which 
implements extra code as needed. This wrapped version decorates the existing 
function. 

But there’s a problem with this, as doing the wrapping on its own is not enough; 
the calling charaderistics of the decorated function need to be maintained, too. 

This means, for instance, that if your existing function accepts two arguments, 
your wrapped function also has to accept two arguments. If you could know 
ahead of time how many arguments to expect, then you could plan accordingly. 
Unfortunately, you can’t know this ahead of time because your decorator can be 
applied to any existing function, which could have—quite literally—any number 
and type of arguments. 

What to do? The solution is to go “generic,” and arrange for the wrapper 
function to support any number and type of arguments. You already know how to 
do this, as you Ve already seen what *args and **kwargs can do. 


Rememker: *args 
and **kwargs 
support any 
number and type 
ol arguments. 



Let's adjust the wrapper function to accept any number and type of 
arguments. Let's also ensure that when fune is invoked, it uses the same 
number and type of arguments as were passed to wrapper. Add in the 
argument code in the spaces provided below: 


VVV^a-t do you 
r\ccd *to add *to 
•tbc 'Vappe/ 

siytftwre? 


def 



check_logged_in(fune): 

def wrapper( ) : 

if 'logged_in' in session: 
return fune( 

return 'You are NOT logged in.' 
return wrapper 


) 


you are here ► 
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a complete decorator 


WA / «7" 

i v Solution 


You were to adjust the wrapper function to accept any number and type 
of arguments, as well as ensure that, when fune is invoked, it uses the 
same number and type of arguments as were passed to wrapper: 


Us'm$ a \C simature 
docs 'britk kcv-c, as 'rt 
suffov-b a*y nuimbev- a*d 
o-f a>r^tY\is. Note 
how wc mvoke v/itb 

the sa^e av^wwewts supplled 
to 'Wapper , y\o r^attev- 

what they are. 


def check_logged_in(fune): 

def wrapper ( *^ar<JS, ^^kv/ar^s ) : 

if 'logged_in' in session: 

return fune ( ^ar^s, ^^kv/ar^s 

return 'You are NOT logged in.' 
return wrapper 


W e're done...or a re we? 

If you check our decorator-creating recipe, you’d be forgiven for believing that 
we’re done. We are...almost. There are two issues that we stili need to deal with: 
one has to do with all decorators, whereas the other has to do with this specific 
one. 

Let’s get the specific issue out of the way first. As the check_logged_in 
decorator is in its own module, we need to ensure that any modules its code refers 
to are also imported into checker . py. The check_logged_in decorator 
uses session, which has to be imported from Flask to avoid errors. Handling 
this is straightforward, as all you need to do is add this import statement to the 
top of checker . py: 

from flask import session 


The other issue, which affects all decorators, has to do with how functions identify 
themselves to the interpreter. When decorated, and if due care is not taken, a 
function can forget its identity, which can lead to problems. The reason why 
this happens is very technical and a little exotic, and it requires a knowledge of 
Python’s internals that most people don’t need (or want) to know. Gonsequently, 
Python’s Standard library comes with a module that handles these details for you 
(so you need never worry about them). All you have to do is remember to import 
the required module (functools), then call a single function (wraps). 

Perhaps somewhat ironically, the wraps function is implemented as a decorator, 
so you don’t actually call it, but rather use it to decorate your wrapper function 
inside your own decorator. WeVe already gone ahead and done this for you, and 
you’11 find the code to the completed check_logged_in decorator at the top 
of the next page. 


When creatingf 
your own 
decorators, always 
import, then use, 
the "lunctools” 
module’s "wraps" 
lunction. 
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Your Pecorator m AII Its ftlory 

Before continuing, make sure your decorator code exactly matches ours: 


suv-c 

4jo ‘iwyorb 

X St$i\OY\ -fv-OW 

the u flask w 

module- 


Deeovate *the 
W Wapp«r w -Puhetioh 
u/ith the U y/\raps >; 
dcto^rdiijoY' (beih^ suy-e 
■fco pass W 'Puhd w as ay> 
avgumcht). 


checker.py - /Users/paul/Desktop/NewBook/ch10/checker.py (3.5.1) 


from flask import session 
from functools import wraps 


def check_logged_in (fune): 

@wraps(fune) 

def wrapper (*args, **kwargs): 
if 'loggedjin' in session: 

return func(*args, **kwargs) 
return 'You are NOT logged in.’ 
return wrapper 



1 

Ln: 13 Coi: 0 

. 



/"•pwr-t ihe Vaps" 
Tundiion (whiih is 
jpel+ a detoralor) 

nodule fwhiih 
is Pairt of ih e 
; ia»da\rd liblrav-y). 


Now that the checker . py module contains a completed check_logged_in 
function, let 5 s put it to use within simple_webapp . py. Here is the current 
version of the code to this webapp (which we’re showing here over two columns): 


from flask import Flask, session 
app = Flask( name ) 


@app.route('/' ) 
def hello() -> str: 

return 'Helio from the simple webapp. 

@app.route('/pagel') 
def pagel () -> str: 

return 'This is page 1.’ 

@app.route(’/page2’) 
def page2() -> str: 

return 'This is page 2.’ 

@app.route(’/page3’) 
def page3() -> str: 

return 'This is page 3.’ 

@app.route(’/login’) 
def do_login() -> str: 

session[’logged_in’] = True 
return 'You are now logged in.’ 


@app.route(’/logout’) 
def do_logout() -> str: 

session.pop(’logged_in') 
return 'You are now logged out.' 

@app.route(’/status') 
def check_status() -> str: 

if ’logged_in’ in session: 

return 'You are currently logged in. 
return 'You are NOT logged in.' 

app.secret_key = 'YouWillNeverGuess...' 

if _name == '_main_' : 

app.run(debug=True) 


Recall that our goal here is to restrict 
access to the / 'pagel , / 'page2 , and /page3 
URLs, which are currently accessible to 
any user’s browser (based on this code). 
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work that decorator 


Puttiwg Your Pecorator to Work 

Adjusting the simple_webapp . py code to use the check_logged_in 
decorator is not difficult. Here’s a list of what needs to happen: 


The check_logged_in decorator needs to be imported from the 
checker . py module. Adding the required import statement to the 
top of our webapp’s code does the trick here. 


Now that the check_logged_in decorator exists, we no longer have 
any need for the check_status function, so it can be removed from 
simple webapp.py. 


To use the check_logged_in decorator, apply it to any of our 
webapp’s functions using the @ syntax. 


© Use the decorator as required 


© Remove any unnecessary code 


O Import the decorator 


Here’s the code to simple_webapp . py once more, with the three changes 
listed above applied. Note how the / 'pagel , / 'page2 , and /page3 URLs now have 
two decorators associated with them: @app . route (which comes with Flask 
and @check logged_in (which youVe just created): 


Apply a dctonlor io 

e*is-fcihg -Pub^tioh usi* 
the § syh-fcax. 


from flask import Flask, session 
from checker import check_logged_in 
app = Flask( name ) 


@app.route( ' / ' ) 
def hello() -> str: 

return 'Helio from the simple webapp. 

@app.route('/pagel') 

@check_logged_in 
def pagel () -> str: 

return 'This is page 1.' 

@app.route('/page2') 

@check_logged_in 
def page2() -> str: 

return 'This is page 2.' 


@app.route('/page3') 

@check_logged_in 
def page3() -> str: 

return 'This is page 3.' 

@app.route('/login' ) 
def do_login() -> str: 

session['logged_in'] = True 
return 'You are now logged in.' 

@app.route('/logout') 
def do_logout() -> str: 

session.pop('logged_in') 
return 'You are now logged out.' 

app.secret_key = 'YouWillNeverGuess...' 

if _name__ == '_main_' : 

app.run(debug=True) 

p 0 «’t Wt a ?b tKese 

ViiaUi^Wted edits to ^ 
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Tqst DriI/q 


To convince ourselves that our login-checking decorator is working as required, let’s take the 
decorator-enabled version of simple_webapp.py for a spin. 

With the webapp running, use a browser to try to access /pagel prior to logging in. After logging in, 
try to access /pagel again and then, after logging out, try to access the restricted content once more. 
Let’s see what happens: 


0 127.0.0.1:5000 


C D 127.0.0.1:5000 


☆ 


Helio from the simple webapp. 


0 127.0.0.1:5000/page1 




C J 127.0.0.1:5000/page1 


You are NOT logged in. 


webapp, tbe bome 
page appears. 


D 127.0.0.1:5000/login 


V 


C D 127.0.0.1:5000/login 


You are now logged in. 



3. A^essihg Vlogih” 
chablcs aucss io 
vestvieted URLs tov 
youv bvou/sev. 


You are now logged out. 


t 


You log out 
£vom tbc webapp 


2- Tvyihg to 
aucss Vpage/ ;; 
v~csults ih ve-Pusal, 
you ave ho-fc 
logged ih. 


D 127.0.0.1:5000/page1 


☆ 



C D 127.0.0.1:5000/page1 


☆ 


This is page 1. 


|\/ 0 vj that you*- bvov/sev is logged in, 
you tan see "/pagel"—suttess' 


• D 127.0.0.1:5000/logout X 

• 

<- C 

D 127.0.0.1:5000/logout <£? 

= 



D 127.0.0.1:5000/page1 


C b 127.0.0.1:5000/page1 


3 


You are NOT logged in. 


As you ave ho lohgev 
logged in, you ean’t see -the 
resWted eontent on the 
/page/ URL anymov-e- 
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gotta love decorators 


The Peauty of Pecorators 


Take another look at the code for your check_logged_in decorator. Note 
how it abstracts the logic used to check if a user 5 s browser is logged in, putting this 
(potentially complex) code in one place— inside the decorator—and then making 
it available throughout your code, thanks to the @check_logged_in decorator 
syntax: 


# # # checker.py -/Users/paul/Desktop/NewBook/ch10/checker.py (3.5.1) 


from flask import session 

from functools import wraps 

def check_logged_in (fune): 

@wraps(fune) 

def wrapper (*args, **kwargs): 
if 'logged.in' in session: 

return func(*args, **kwargs) 
return 'You are NOT logged in.' 
return wrapper 


Ln: 1 3 Coi: 0 




This todt 
looks Aeaky, 
tui isht redi |y. 


Abstracting code in a decorator makes the code that uses it easier to read. 
Gonsider this usage of our decorator on the /page2 URL: 


# # # simple webapp4.py -/Users/paul/Desktop/NewBook/ch10/simple webapp4.py (3.5.1) 

@app.route( '/page2' ) 

@check logged in 
def page2 () : 

return 'This is page 2.' 

Wsrng a deioraW 
»akes -this code 


easier -to read. 

Ln: 1 Coi: 0 


Note how the page2 function’s code is only concerned with what it needs to do: 
display the /page2 content. In this example, the page2 code is a single, simple 
statement; it would be harder to read and understand if it also contained the logic 
required to check whether a user’s browser is logged in or not. Using a decorator 
to separate out the login-checking code is a big win. 

This “logic abstraction” is one of the reasons the use of decorators is popular in 
Python. Another is that, if you think about it, in creating the check_logged 
in decorator, you Ve managed to write code that augments an existingfunction with 
extra code, hy changing the behavior of the existingfunction without changing its code. When 
it was first introduced earlier in this chapter, this idea was described as “freaky.” 
But, now that you Ve done it, there’s really nothing to it, is there? 


Decorators 
arent Ireaky 
tney Ve lun. 
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Crcatiwg More Pecorators 

With the process of creating the check_logged_in decorator behind you, you 
can use its code as the basis of any new decorators you create from here on in. 

To make your life easier, here 5 s a generic code template (in the file tmpl 
decorator.py) that you can use as the basis of any new decorators you write: 


tmpl decorator.py - /Users/paul/Desktop/NewBook/ch10/tmpl decorator.py (3.5.1) 


from functools import wraps 

def decorator_name (fune): 

@wraps(fune) 

def wrapper (*args, **kwargs): 

# 1. Code to exeeute BEFORE calling the decorated function. 

# 2. Call the decorated function as required, returning its 

# results if needed. 

return func(*args, **kwargs) 

# 3. Code to exeeute INSTEAD of calling the decorated function 
return wrapper 



Ln: 1 5 Coi: 0 


This code template can be adjusted as needed to suit your needs. All you need 
to do is give your new decorator an appropriate name, then replace the three 
comments in the template with your decoratods specific code. 

If it makes sense for your new decorator to invoke the decorated function without 
returning its results, that 5 s fine. After all, what you put in your wrapper function 
is your code, and you are free to do whatever you want to. 


Replade these 
tommemis with youv 

£ode. 


tlierei are no 

Diimb Qtiestions 



Arerft decorators just like the last chapteds context manager in that they both let me wrap code with additional functionality? 


Thafs a great question. The answer is: yes and no. Yes, both decorators and context managers augment existing code with additional 
logic. But no, they are not the same. Decorators are specifically concerned with augmenting existing functions with additional functionality, 
whereas context mangers are more interested in ensuring your code exeeutes within a specific context, arranging for code to run before a 
with statement as well as ensuring that code always exeeutes after a with statement. You can do something similar with decorators, 
but most Python programmers would regard you as a little mad if you were to attempt this. Also, note that your decorator code is under no 
obligation to do anything after it invokes the decorated function (as is the case with the check_logged_in decorator, which does 
nothing). This decorator behavior is very different from the protocol that context managers are expected to adhere to. 


you are here ► 


407 











back to vsearch4web.py 


Pack to Restrictiwg Access to /viewlog 



Now that youVe created a mechanism that lets you restrict access to certain URLs 
in simple_webapp . py, it’s a no-brainer to apply the same mechanism to any 
other webapp. 

This includes vsearch4web . py, where you had a requirement to restrict access 
to the /viewlog URL. All you need to do is copy the do_login and do_logout 
functions from simple_webapp .py into vsearch4web .py, import the 
checker . py module, and then decorate the view_the_log function with 
check_logged_in. Granted, you may want to add some sophistication to 
do_login and do_logout (by, perhaps, checking user credentials against those 
stored in a database), but—as regards restricting access to certain URLs—the 
check_logged_in decorator does most of the heavy lifting for you. 
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What'$ Next? 

Rather than spend a bunch of pages doing to vsearch4web . py what youVe 
just spent a chunk of time doing to simple_webapp . py, we’re going to leave 
adjusting vsearch4web . py for you to do onyour own. At the start of the next 
chapter, weTl present an updated version of the vsearch4web . py webapp for 
you to compare with yours, as our updated code is used to frame the discussion in 
the next chapter. 

To date, all of the code in this book has been written under the assumption that 
nothing bad ever happens, and nothing ever goes wrong. This was a deliberate 
strategy on our part, as we wanted to ensure you had a good grasp of Python 
before getting into topics such as error correction, error avoidance, error detection, 
exception handling, and the like. 

We have now reached the point where we can no longer follow this strategy The 
environments within which our code runs are real, and things can (and do) go 
wrong Some things are fixable (or avoidable), and some arerft. If at all possible, 
yoirll want your code to handle most error situations, only resulting in a crash 
when something truly exceptional happens that is beyond your control. In the next 
chapter, we look at various strategies for deciding what 5 s a reasonable thing to do 
when stuff goes wrong. 

Prior to that, though, here 5 s a quick review of this chaptebs key points. 



BULLET POINTS 


■ When you need to store server-side state within a Flask 
webapp, use the session dictionary (and don’t forget 
to set a hard-to-guess secret_key ). 

■ You can pass a function as an argument to another 
function. Using the function’s name (without the 
parentheses) gives you a function object, which can be 
manipulated like any other variable. 

■ When you use a function object as an argument to a 
function, you can have the receiving function invoke the 
passed-in function object by appending parentheses. 

■ A function can be nested inside an enclosing function’s 
suite (and is only visible within the enclosing scope). 

■ In addition to accepting a function object as an 
argument, functions can return a nested function as a 
return value. 


*args is shorthand for “expand to a list of items.” 

**kwargs is shorthand for “expand to a dictionary of 
keys and values.” When you see “kw,” think “keywords.” 

Both * and ** can also be used “on the way in,” in that 
a list or keyword collection can be passed into a function 
as a single (expandable) argument. 

Using (*args, **kwargs) as a function signature 

lets you create functions that accept any number and 
type of arguments. 

Using the new function features from this chapter, you 
learned how to create a function decorator, which 
changes the behavior of an existing function without the 
need to change the function’s actual code. This sounds 
freaky, but is quite a bit of fun (and is very useful, too). 


you are here ► 
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Chapter 10'$ Code, 1 of 2 


TWis is h 

"<\i*itk_sessionf/ • 



from flask import Flask, session 
app = Flask( name ) 


app.secret_key = 'YouWillNeverGuess' 

@app.route('/setuser/<user>') 
def setuser(user: str) -> str: 
session['user 1 ] = user 

return 'User value set to: ' + session[ 1 user 1 ] 

@app.route('/getuser') 
def getuser() -> str: 

return 'User value is currently set to: ’ + session[’user’] 

if _name = ’_main_' : 

app.run(debug=True) 


from flask import session 

from functools import wraps 

def check_logged_in(func): 

@wraps(fune) 

def wrapper(*args, **kwargs): 
if 'logged_in' in session: 

return func(*args, **kwargs) 
return 'You are NOT logged in.' 
return wrapper 


This Wker.py", whidh dwiaihs 
the todt to -this £hap-te^s 
decorator: ^he^kjogged_> w 



This is u 't^pLde6ov-a-tov- p/> ^hith 
is a ha*dy defioralor-dv-eatmj 
•templa-te $or you tx> v-euse as you 
see -fit 


Chapter 10 


from functools import wraps 

def decorator_name(fune): 
fwraps(fune) 

def wrapper(*args, **kwargs): 

# 1. Code to exeeute BEFORE calling the decorated function. 

# 2. Call the decorated function as required, returning its 

# results if needed. 

return func(*args, **kwargs) 

# 3. Code to exeeute INSTEAD of calling the decorated function. 
return wrapper 
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Chapter 10'$ Code, 2 of 2 


TWis is ''simylejwebaypfy > '"bidb 

?u IU all oi ibis dbapieA e-odc 

Wtbev. Men 70« need io 
vestriti addess io syed.f >d URU 
base 70WV sirate57 on ibis 

Y/cbap^ s w\C6bay\iSr»\. 



I^Vc think “thc use of 
dedo^aWs makes this 
webapps dode easy t. 
<read and «hdevsfcand- 

Don'i you? © 



from flask import Flask, session 
from checker import check_logged_in 
app = Flask( name ) 


@app.route {'/') 
def hello() -> str: 

return 'Helio from the simple webapp.' 

@app.route('/pagel') 

@check_logged_in 
def pagel() -> str: 

return 'This is page 1.' 

@app.route('/page2') 

@check_logged_in 
def page2() -> str: 

return 'This is page 2.' 

@app.route('/page3') 

@check_logged_in 
def page3() -> str: 

return 'This is page 3.' 

@app.route('/login') 
def do_login() -> str: 

session['logged_in'] = True 
return 'You are now logged in.' 

@app.route('/logout') 
def do_logout() -> str: 

session.pop('logged_in') 
return 'You are now logged out.' 

app.secret_key = 'YouWillNeverGuessMySecretKey' 

if _name_ == '_main_' : 

app.run(debug=True) 


you are here ► 
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11 exception Kandlfng 



Things go wrong, all the time—no matter how good your code is. 

You’ve successfully executed all of the examples in this book, and yoiTre likely confident all 
of the code presented thus far works. But does this mean the code is robust? Probably not. 
Writing code based on the assumption that nothing bad ever happens is (at best) naive. At 
worst, it’s dangerous, as unforeseen things do (and will) happen. It’s much better if you’re 
wary while coding, as opposed to trusting. Care is needed to ensure your code does what 
you want it to, as well as reacts properly when things go south. In this chapter, you’ll not only 
see what can go wrong, but also learn what to do when (and, oftentimes, before) things do. 


this is a new chapter 


413 








looking for issues 




We’re starting this chapter by diving right in. Presented below is the latest code to the 
vsearch4web. py webapp. As you’11 see, we’ve updated this code to use the check_ 
logged_in decorator from the last chapter to control when the information presented by the 
/viewlog URL is (and isn’t) visible to users. 

Take as long as you need to read this code, then use a pencil to circle and annotate the parts 
you think might cause problems when operating within a production environment. Highlight 
everything that you think might cause an issue, not just potential runtime issues or errors. 


from flask import Flask, render_template, request, escape, session 
from vsearch import search41etters 

from DBcm import UseDatabase 

from checker import check_logged_in 

app = Flask( name ) 


app.config['dbconfig'] = {'host': '127.0.0.1', 

'user': 'vsearch', 

'password': 'vsearchpasswd', 

'database': 'vsearchlogDB', } 

@app.route('/login') 
def do_login() -> str: 

session['logged_in'] = True 
return 'You are now logged in.' 

@app.route('/logout') 
def do_logout() -> str: 

session.pop('logged in') 
return 'You are now logged out.' 

def log_reguest(req: 'flask_request', res: str) -> None: 

with UseDatabase(app.config['dbconfig']) as cursor: 

SQL = """insert into log 

(phrase, letters, ip, browser_string, results) 
values 

/ 9. o 9-o 9-0 9-o 9 0 \ n n m 
i ob, oS, oS, oS, ob| 

cursor.execute( SQL, (req.form['phrase'], 

req.form['letters'], 
req.remote_addr, 
req.user_agent.browser, 
res, )) 
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@app.route('/search4', methods=['POST']) 
def do_search() -> 'html': 

phrase = request.form['phrase'] 

letters = request.form['letters'] 

title = 'Here are your results:' 

results = str(search41etters (phrase, letters)) 

loq_request(request, results) 

return render_template(’results.html’, 

the_title=title, 
the_phrase=phrase, 
the_letters=letters, 
the_results=results ,) 

@app.route('/’) 

@app.route('/entry') 

def entry paqe() -> 'html’: 

return render_template('entry.html', 

the title='Welcome to search41etters on the web!') 


@app.route('/viewloq') 

@check_loqqed_in 

def view_the_loq() -> 'html': 

with UseDatabase(app.confiq['dbconfiq']) as cursor: 

SQL = """select phrase, letters, ip, browser_strinq, results 
from loq""" 
cursor.execute(_SQL) 
contents = cursor.fetchall() 

tities = ('Phrase', 'Letters', 'Remote_addr', 'User_aqent', 'Results') 

return render_template('viewloq.html', 

the_title='View Loq', 
the_row_titles=titles, 
the data=contents,) 


app.secret_key = 'YouWillNeverGuessMySecretKey' 

if _name_ == '_main_' : 

app.run(debuq=True) 


you are here ► 


415 









identifying issues 




yiiOH 


You were to take as long as you needed to read the code shown below (which is an updated 
version of the vsearch4web. py webapp). Then, using a pencil, you were to circle and 
annotate the parts you thought might cause problems when operating within a production 
environment. You were to highlight everything you thought might cause an issue, not just 
potential runtime issues or errors. (We’ve numbered our annotations for ease of reference.) 


from flask import Flask, render_template , request, escape, session 
from vsearch import search41etters 

from DBcm import UseDatabase 



req.user_agent.browser, 
res, )) 
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exception handling 


@app.route ( '/search4', methods=['POST']) 
def do_search() -> 'html': 

phrase = request.form['phrase'] 
letters = request.form['letters ! ] 
title = 'Here are your results:’ 
resulta - ~ J ~- 7 / p hragQ J 



letters)) 


.ntml’, 
the_title=title, 
the_phrase=phrase, 
the_letters=letters, 
the results=results,) 


happchs 

dall +ails? 


@app.route(’/') 

@app.route(’/entry’) 

def entry paqe () -> ’html’ : 

return render_template(’entry.html’, 

the title=’Welcome to search41etters on the web!’) 


@app.route(’/viewloq’) 

@check_loqqed_in 

def view loq () -> 'html’: 

wit/n Useb^tabase (app . conf iq [ ’ dbconf iq ’ ] ) as cursor: 

select phrase, letters, ip, browserstring, results 

L J OiLL " i v 



_ II II II 


II llv' 



cursor.execute(_SQL) 
contents = cursor.fetchall() 
tities = ('Phrase', 'Letters', 'Remote_addr', 
return render_template('viewloq.html', 

the_title='View Loq', 
the_row_titles=titles, 
the data=contents,) 


'User aqent', 'Results') 


app.secret_key = 'YouWillNeverGuessMySecretKey' 

if _name_ == '_main_' : 

app.run(debuq=True) 


you are here ► 
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the issues 


Patabases Arent Always Available 

WeVe identified four potential issues with the vsearch4web . py code, and we 
concede that there may be many more, but we’11 worry about these four issues for now. 
Let 5 s consider each of the four issues in more detail (which we do here and on the next 
few pages, by simply describing the problems; we 3 ll work on Solutions later in this chapter). 
First up is worrying about the backend database: 



What happens if the database connection fails? 

Our webapp blissfully assumes that the backend database is always 
operational and available, but it may not be (for any number of reasons). 
At the moment, it is unclear what happens when the database is down, 
as our code does not consider this eventuality. 


Let’s see what happens if we temporarily switch off the backend database. As you can 
see below, our webapp loads fine, but as soon as we do anything, an intimidating error 
message appears: 


12) Welcome to search4letter X 


C Q 127.0.0.1:5000 


Evev-ytWm^ 
looks 0^ 
heve- 




Welcome to search4letters on the web! 


Use this form to sublinit a search request: 


Phrase: Life, the Universe, and Everything 
Letters: aeiou 


When you're ready, click this button: 


Do it! 


| mysql.connector.errors.lnt X 


C Q 127.0.0.1:5000/search4 


☆ 


mysql.connector.errors.InterfaceError 

mysql.connector.errors.InterfaceError: 2003: Can't connect to MySQL server on '127.0.0.1:3306' (61 
Connection refused) 


Traceback (most recent call last) 


File 7Library/Frameworks/F’ython.framework/Versions/3.5/lib/python3.5/site-packages/flask/app.py", line 2000 , 
in_call_ 

return self,wsgi_app(environ, start_response) 

File 7Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/flask/app.py", line 1991 , 
in wsgi_app 

response = self.make_response(self.handle_exception(e)) 

File 7Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/flask/app.py", line 1567 , 
in handle_exception 


<r 


A 


...kwti wbe» you tlidk 
on Lbe "po rt!” builon, 
Lke vwebayp trasbes wtb 
a« "Inlev-tadeEvv-ov”. 
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exception handling 


Web Attacks Are a Real Pain 

As well as worrying about issues with your backend database, you also need to 
worry about nasty individuals trying to do nasty things to your webapp, which 
brings us to the second issue: 

o Is our webapp protected from web attacks? 

The phrases SQL injection (SQLi) and Cross-site scripting (XSS) should 
strike fear in the heart of every web developer. The former allows 
attackers to exploit your backend database, while the latter allows them 
to exploit your website. There are other web exploits that you’11 need to 
worry about, but these are the “big two.” 

As with the first issue, let 5 s see what happens when we try to simulate these exploits 
against our webapp. As you can see, it appears we 5 re ready for both of them: 

[t you hy to inj td SQL m-fco 


Welcome to search4letter x 


V D 

C D 127.0.0.1:5000 


■the web ihtev-Paie, it has „„ 

.««rated 


Welcome to search4letters on the web! 

Use this form to submit a search request: 



/Wy atterat to exploit XSS by ^edm 9 JavaSevi ? t 
to the webapp has «o e-ftefrt 



☆ 


X ,fa U fc tt e 


Welcome to search4letters on the web! 

Use this form to|/jbmit a search request: 


••• 

^ Here are your results: x 


• 

«■ c 

Q 127.0.0.1:5000/search4 

1 ☆! 

— 


Phrase: <script type='text/javascript'>alert( , Hello! , );</script> 
Letters: aeiou 


When you're ready, click this button: 


Here are your results' 

You submitted the following data: 


Phrase: 

<script type='text/t 

/,vascript , >alert('Hello!’);</script> 

Letters: 

aeiou J 



When "<script type= , text/javascript'>alert( , Helio!');</script>" is search for 
"aeiou", the following results are returned: 

{'a', 'e', 'i', 'o'> 





































































more issues 


Input-Output 1$ (Sometimes) Slow 

At the moment, our webapp communicates with our backend database in an 
almost instantaneous manner, and users of our webapp notice little or no delay as 
the webapp interacts with the database. But imagine if the interactions with the 
backend database took some time, perhaps seconds: 


Perhaps the backend database is on another machine, in another 
building, on another continent...what would happen then? 


© What happens if something takes a long time? 


Communications with the backend database may take time. In fact, whenever 
your code has to interact with something that’s external to it (for example: a file, a 
database, a network, or whatever), the interaction can take any amount of time, 
the determination of which is usually beyond your control. Despite this lack of 
control, you do have to be cognizant that some operations may be lengthy. 

To demonstrate this issue, let 5 s add an artificial delay to our webapp (using the 
sleep function, which is part of the Standard library 5 s time module). Add this 
line of code to the top of your webapp (near the other import statements): 

from time import sleep 


With the above import statement inserted, edit the log_request function 
and insert this line of code before the with statement: 


sleep(15) 


If you restart your webapp, then initiate a search, there’s a very distinet delay 
while your web browser waits for your webapp to catch up. As web delays go, 15 
seconds will feel like a lifetime, which will prompt most users of your webapp to 
believe something has crashed : 


Welcome to search4letters o 
X :E : © 127.0.0.1:5000 


+ 


i 


Welcome to search4letters on the web! 


$tev 

ii!" Wfcb». Y w " r ** ., 

bvoY/sev Wrb -and ' Na, ‘ ts " 
a«di yiarb-and *iavb••• 



Use this form to submit a search request: 


Phrase: 

This search takes 15 seconds 


Letters: 

aeiou 



When you're ready, click this button: 


Do it! 


Waiting for 127.0.0.1... 
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exception handling 


Your Function Calis Can Fail 

The final issue identified during this chapter’s opening exercise relates to the 
function call to log re que st within the do search function: 



What happens if a function call fails? 

There’s never a guarantee that a function call will succeed, especially if 
the function in question interacts with something external to your code. 


WeVe already seen what can happen when the backend database is unavailable 
the webapp crashes with an Interf aceError: 



dalabase, so 
youv webapp 
dvasbcs. 



Other problems can surface, too. To simulate another error, find the sleep (15) 
line you added from the Issue 3 discussion, and replace it with a single statement: 
raise. When executed by the interpreter, raise forces a runtime error. If you 
try your webapp again, a different error occurs this time: 



a*d 

you\r webapp 
£>-ashes ag a i h . 


*r 



I RuntimeError: No active ei 




127.0.0.1:5000/search4 


☆ 


builtins.RuntimeError 

RuntimeError: No active exception to reraise 


Traceback (most recent call last) 


File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/flask/app.py", line 
2000 , in_call_ 

return self.wsgi_app(environ, start_response) 

File 7Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/flask/app.py" 1 line 
1991 , in wsgi_app 

response = self.make_response(self.handle_exception(e)) 


File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/flask/app.py", line 
1567 , in handle_exception 



r * 

Be-fore “tbc 

pa^e, vcw^ove “tbat 
dall bo Vaisc" &0* 
youv dode “to 
-the webapp s*tav“U 
wovkm^ a^arn. 


you are here ► 
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what to do? 


Cowsidermg the Identif ied Problews 

WeVe identified four issues with the vsearch4web . py code. Let’s revisit each 
and consider our next steps. 


1. Your database connectum fails 


Errors occur whenever an external System your code relies on is unavailable. The 
interpreter reported an InterfaceError when this happened. It’s possible to 
spot, then react to, these types of errors using Python 5 s built-in exception-handling 
mechanism. If you can spot when an error occurs, you’re then in a position to do 
something about it. 

2 . Your application is subjected to an attack 

Although a case can be made that worrying about attacks on your application 
is only of concern to web developers, developing practices that improve 
the robustness of the code you write are always worth considering. With 
vsearch4web . py, dealing with the “big two” web attack vectors, SQL injection 
(SQLi) and Cross-site scripting (XSS), appears to be well in hand. This is more of 
a happy accident than by design on your part, as the Jinja2 library is built to 
guard against XSS by default, escaping any potentially problematic strings (recall 
that the JavaScript we tried to trick our webapp into executing had no effect). As 
regards SQLi , our use of DB-APEs parameterized SQL strings (with all those ? 
placeholders) ensures—again, thanks to the way these modules were designed— 
that your code is protected from this entire class of attack. 

3. Your code takes a long time to execute 

If your code takes a long time to execute, you have to consider the impact on your 
user 5 s experience. If your user doesn’t notice, then you’re likely OK. However, if 
your user has to wait, you may have to do something about it (otherwise, your user 
may decide the wait isn’t worth it, and go elsewhere). 



If you want to know 
more about SQLi and 
XSS, Wikipedia is a great 
place to start. See https:// 
en. wikipedia. org/wiki/ 
SQLJnjection and https:// 
en. wikipedia. org/wiki/Cross- 
site_scripting, respectively. 
And remember, there are 
all kinds of other types 
of attack that can cause 
problems for your app; 
these are just the two 
biggies. 


4. Your function call fails 

It 5 s not just external Systems that generate exceptions in the interpreter—your 
code can raise exceptions, too. When this happens, you need to be ready to spot 
the exception, then recover as needed. The mechanism you use to enable this 
behavior is the same one hinted at in the discussion of issue 1, above. 

So...where do we start when dealing with these four issues? It’s possible to use the 
same mechanism to deal with issues 1 and 4, so that’s where wedl begin. 
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Always Try to Execute Error-Prowe Code 

When something goes wrong with your code, Python raises a runtime exception. 
Think of an exception as a controlled program crash triggered by the interpreter. 

As youVe seen with issues 1 and 4, exceptions can be raised under many different 
circumstances. In fact, the interpreter comes with a whole host of built-in 
exception types, of which RuntimeError (from issue 4) is only one example. 

As well as the built-in exception types, it 5 s possible to define your own custom 
exceptions, and youVe seen an example of this too: the InterfaceError 
exception (from issue 1) is defined by the MySQL Connector module. 

To spot (and, hopefully, recover from) a runtime exception, deploy Python’s try 
statement, which can help you manage exceptions as they occur at runtime. 

To see try in action, let’s first consider a snippet of code that might fail when 
executed. Here are three innocent-looking, but potentially problematic, lines of 
code for you to consider: 


For a complete list of tlie 
tui It -in exceptions, see 
https'/ / ctocs.pytkon.orgf/3/ 
libraiy / exceptions Jktml. 


try examples.py - /Users/paul/Desktop/NewBo. 


■> 


Theres wiWmj 'weivd ov 'wondevtul 

ojomo) on ber* ibe na«>ed f'le >s 
o ?e ^d, a«d iis data is oWamed a*d 
ihe« displayed oh Street- 


with open (' myf ile . txt' ) as fh: 

file_data = fh.read() 
print (file_data) 

Ln: 5 Coi: 0 


There 5 s nothing wrong with these three lines of code and—as currently written- 
they will execute. However, this code might fail if it can’t access mytile . txt. 
Perhaps the file is missing, or your code doesnl have the necessary file-reading 
permissions. When the code fails, an exception is raised: 


Meh «3 Iruhtimc erv-ov- 
Pytho, displays 

a "tradebadk", whidh 
details what weht 

WV"Oh(j, 

Iu this case, the 
mievpreier ihihks the 
pvoblem is oh line 2. 


Wboops! 


I 



Python 3.5.1 Shell 



»> 

========= RESTART: /Users/paul/Desktop/_NewBook/chll/try_examples.py =? 

Traceback (most recent call last) : 

File "/Users/paul/Desktop/_NewBook/chll/try_examples.py", line 2, in <module> 
with open('myfile.txt' ) as fh: 

FileNotFoundError: [Errno 2] No such file or directory: 'myfile.txt' 

»> 

»> 

fh: 119 Coi: 4 


Let 5 s start learning what try can do by adjusting the above code snippet to 
protect against this FileNotFoundError exception. 



Desyrte bcmj u^ly *to look 
ai, the ivadebatk messa^e is 
usetul- 

you are here ► 423 

























try try try 


Catching an Error Is Not Enough 


When a runtime error occurs, an exception is raised. If you ignore a raised exception 
it is referred to as uncaught, and the interpreter will terminate your code, then 
display a runtime error message (as shown in the example from the bottom of the last 
page). That said, raised exceptions can also be caught (i.e., dealt with) with the try 
statement. Note that its not enough to catch runtime errors, you also have to decide 
what yoffre going to do next. 

Perhaps yoffll decide to deliberately ignore the raised exception, and keep going...with 
your fingers firmly crossed. Or maybe yoffll try to run some other code in place of the 
code that crashed, and keep going. Or perhaps the best thing to do is to log the error 
before terminating your application as cleanly as possible. Whatever you decide to do, 
the try statement can help. 

In its most basic form, the try statement allows you to react whenever the execution 
of your code results in a raised exception. To protect code with try, put the code 
within try’s suite. If an exception occurs, the code in the try’s suite terminates, and 
then the code in the try’s except suite runs. The except suite is where you define 
what you want to happen next. 

Lets update the code snippet from the last page to display a short message whenever 
the FileNotFoundError exception is raised. The code on the left is what you had 
previously, while the code on the right has been amended to take advantage of what 
try and except have to offer: 


When a runtime 
error is raised, it 
can be cautfbt or 
uncaugfbt: try 
lets you catch a 
raised error, and 
"except” lets you 
do sometbing 
about it. 



Note how the en-tive tode 
shiffct is mdented vndev- 

i\\t w *bY 


O O try_examples.py -/Users/paul/Desktop/\^iewBo 

with open (' myfile.txt' ) as fh: 

file_data = fh.read() 
print (file_data) 

Ln: 5 Coi: 0 



The exeepi s-ta-tcmeht is 
mdeivfced -to the sa*>e leve! 
3s its assodiated u tvy ;; ; 

3fid has its ovm suite* 



try examples2.py - /Users/paul/Desktop/NewBook/ch11 



try : 

with open ( 'myfile.txt' ) as fh: 

file_data = fh.read() 
print (file_data) 
except FileNotFoundError: 

print (' The data file is missing.') 


Ln: 8 Coi: 0 



Note that what was three lines of code is now six, which may seem wasteful, but isfft. 
The original snippet of code stili exists as an entity; it makes up the suite associated 
with the try statement. The except statement and its suite is new code. Lets see 
what difference these amendments make. 


■TWis toAc is mdenLed 

the Vte ? t” tU* 

a»d o«ly e*etutes it ^ 
the "PileNotFowd&rvor 
ei^eeftio*' is raised- 
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Tqst Drwq 


Let’s take the try...except version of your code snippet for a spin. If myf ile. txt exists and 
is readable by your code, its contents will appear on screen. If not, a run-time exception is raised. 
We already know that myf ile. txt does not exist, but now, instead of seeing the ugly traceback 
message from earlier, the exception-handling code fires and we’re presented with a friendlier 
message (even though our code snippet stili crashed): 

The -fivst We you va* the tode sniyyet, the 
mtevf vetev genev-ated this ugly tradebadk. - 


»> 



Python 3./>.1 Shell 


========= RESTART: /Users/paul/Desktop/_NewBook/chll/try_examples.py 

Traceback (most recent call last): 

File "/Users/paul/Desktop/_NewBook/chll/try_examples.py", line 2, in <module> 
with open('myfile.txt') as fh: 

FileNotFoundError: [Errno 2] No such file or directory: 'myfile.txt' 

»> 

»> 

======== RESTART: /Users/paul/Desktop/_NewBook/chll/try_examples2.py ======== 

The data file is missing. 

>>> I 

Ln: 1 7 Coi: 4 



The new vev-sion o-f *the £ode pv-odudes 
a mudh tviendlier messays thanks to 
"tvy" and exteyt • 


There caH be more thaw owe exception raised... 

This new behavior is better, but what happens if myf ile . txt exists but your 
code does not have permission to read from it? To see what happens, we created 
the file, then set its permissions to simulate this eventuality. Rerunning the new 
code produces this output: 


W WeVe back to seeing 
an ugly tnadebadk message, 
as a "Pev-missionEv-v-ov-" w as 
raised. 


# # # Python 3.5.1 Shell 

»> 

======== RESTART: /Users/paul/Desktop/_NewBook/chll/try_examples2.py ==== === 

The data file is missing. 

»> 

======== RESTART: /Users/paul/Desktop/_NewBook/chll/try_examples2.py = 

Traceback (most recent call last) : 

File M /Users/paul/Desktop/_NewBook/chll/try_examples2.py M , line 3, in <module> 
with open('myfile.txt') as fh: 

PermissionError: [Errno 13] Permission denied: 'myfile.txt' 

»> 

»> | 

Ln: 24 Coi: 4 



you are here > 
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multiple excepts 


try Once, but except Many Times 

To protect against another exception being raised, simply add another except 
suite to your try statement, identifying the exception you’re interested in and 
providing whatever code you deem necessary in the new except’s suite. Here’s 
another updated version of the code that handles the PermissionError 
exception (should it be raised): 


|n addrtion "to y 
TileKo-tFowrxlBvvov 

tWis toAc al so w 

Ka^dles a "Pe<r«.issto*»&rror • 



try examples3.py - /Users/paul/Desktop/NewBook/ch11/try. 


try : 

with open (' myfile.txt' ) as fh: 

file_data = fh.read() 
print (file_data) 
except FileNotFoundError : 

print (' The data file is missing.') 
except PermissionError: 

print ('This is not allowed.') ^ 



Ln: 1 0 Coi: 0 



The Code ih the 
exeept suites 


Executing this amended code stili results in the PermissionError exception 
being raised. However, unlike before, the ugly traceback has been replaced by a much 
friendlier message: 


C2y\ do dhy-fchihg. 
Fov how, eaeh 
displays a 

-pHehdly message- 


# # # Python 3.5.1 Shell 

»> 

======== RESTART: /Users/paul/Desktop/_NewBook/chll/try_examples2.py ======== 

Traceback (most recent call last) : 

File M /Users/paul/Desktop/_NewBook/chll/try_examples2.py M , line 3, in <module> 
with open('myfile.txt') as fh: 

PermissionError: [Errno 13] Permission denied: 'myfile.txt' 

»> 

»> 


RESTART: /Users/paul/Desktop/_NewBook/chll/try_examples3.py 



This is looking good: you Ve managed to adjust what happens whenever the file you’re 
hoping to work with isn’t there (it doesn’t exist), or is inaccessible (you doni have the 
correct permissions). But what happens if an exception is raised that you weren’t 
expecting? 
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exception handling 


A lot of Thiwgs Caw fto Wrowg 

Before answering the question posed at the bottom of the last page— what happens if 
an exception is raised that you werenP expecting? —take a look at some of Python 3 5 s built- 
in exceptions (which are copied directly from the Python documentation). Don’t be 
surprised if you’re struck by just how many there are: 


Exception 



AII the bwlt— 

1 

'm eK-te\>t'o»s 

+ -- 

'mhevit t 'ror* 

+ -- 

a tlass talled 

+ -- 

on • 

+ -- 

+ -- 


+-- 


+ -- 


+ -- 


+ -- 



"there? 


Stoplteration 
StopAsyncIteration 
ArithmeticError 
+-- FloatingPointError 
+-- OverflowError 
+-- ZeroDivisionError 
AssertionError 
AttributeError 
BufferError 
EOFError 
ImportError 
LookupError 
+-- IndexError 
+-- KeyError 
MemoryError 
NameError 

+-- UnboundLocalError 
OSError 

+-- BlockinglOError 
+-- ChildProcessError 
+-- ConnectionError 
| +-- BrokenPipeError 

| +-- ConnectionAbortedError 

| 4 - ConnectionRefusedError 

| +-- ConnectionResetError 

+-- FileExistsError 
+-- FileNotFoundError 
+-- InterruptedError 
+-- IsADirectoryError 
+-- NotADirectoryError 
+-- PermissionError 
+-- ProcessLookupError 
+-- TimeoutError 


Were s *the "two 
«ceptiohs -tha-t our 
tode Cuv-yrem-tly handles. 


It would be crazy to try to write a separate except suite for each of these runtime 
exceptions, as some of them may never occur. That said, some might occur, so you 
do need to worry about them a little bit. Rather than try to handle each exception 
individually , Python lets you deflne a catch-all except suite, which flres whenever a 
runtime exception occurs that you haven’t speciflcally identifled. 


you are here ► 
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catching all exceptions 


The Catch-AII Exception Handler 

Let 5 s see what happens when some other error occurs. To simulate just such an 
occurrence, weVe changed myf ile . txt from a file into a folder. Let’s see what 
happens when we run the code now: 


Python 3.5.2 Shell 


»> 


======== RESTART: /Users/paul/Desktop/_NewBook/chll/try_examples3.py ======== 

Traceback (most recent call last): 

File "/Users/paul/Desktop/_NewBook/chll/try_examples3.py", line 3, in <module> 
with open('myfile.txt') as fh: 

IsADirectoryError: [Errno 21] Is a directory: 'myfile.txt' ^ 

»> 

5 Coi: 4 


Another exception is raised. You could create an extra except suite that fires when 
this IsADirectoryError exception occurs, but let’s specify a catch-all runtime 
error instead, which fires whenever any exception (other than the two weVe already 
specified) occurs. To do this, add a catch-all except statement to the end of the 
existing code: 

# # # try_examples4.py - /Users/paul/Desktop/_NewBook/ch11/tr. 


This 

statc^c^t 
is W ,; vt 

does Y\oi refer 

{o a spe£i-fi£ 


■\ oy \- 


try : 

with open (' myfile.txt' ) as fh: 

file_data = fh.read() 
print (file_data) 
except FileNotFoundError : 

print ('The data file is missing.') 
except PermissionError : 

print ('This is not allowed.') 
except : 

print ('Some other error occurred.' ) 


Ln: 12 Coi: 0 



Aho-fchev 

has o£duirv-ecl. 


This toAt provides 

a eaUh-all 

c^eeption handler. 



Running this amended version of your code gets rid of the ugly traceback, displaying 
a friendly message instead. No matter what other exception occurs, this code handles it 
thanks to the addition of the catch-all except statement: 


This looks 
betfceir. 


Python 3.5.2 Shell 


»> 

======== RESTART: /Users/paul/Desktop/_NewBook/chll/try_exfmples3.py 

Traceback (most recent call last): 

File "/Users/paul/Desktop/_NewBook/chll/try_examples3.]//", line 3, in <module> 
with open('myfile.txt') as fh: 

IsADirectoryError: [Errno 21] Is a directory: 'myfile.t/.t' 

»> 

======== RESTART: /Users/paul/Desktop/_NewBook/chll/£^y_examples4.py 

Some other error occurred. 

»> I 




Ln: 16 Coi: 4 
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exception handling 


Haven't We Just lost Sowethiwg? 


0 



OK. I get wha+'s going on here. But 
does this code not now hide the fact that 
we just had an "IsADirectoryError"? Isn't 
it important to know exactly which error 
youve encountered? 


Ah, yes...good catch. 

This latest code has tidied up the output (in 
that the ugly traceback is gone), but youVe 
also lost some important information: you no 
longer know what the specific issue with your 
code was. 

Knowing what exception was raised is often 
important, so Python lets you get at the data 
associated with the most-recent exception 
information as ifs being handled. There are two 
ways to do this: using the facilities of the sys 
module, and using an extension to the try/ 
except syntax. 

Let’s look at both of these techniques. 


_t]i©r©7£tr©jio 

Dumb Questipns 

Is it possible to create a catch-all exception handler that does nothing? 

Yes. It is often tempting to add this except suite to the bottom of a try statement: 

except: 
pass 



Please try not to do this. This except suite implements a catch-all that ignores any other exception (presumedly in the misguided hope 
that if something is ignored it might go away). This is a dangerous practice, as—at the very least—an unexpected exception should resuit in an 
error message appearing on screen. So, be sure to always write error-checking code that handles exceptions, as opposed to ignores them. 


you are here ► 


429 








what went wrong? 


learwmg About Exccptiows f row "sys" 

The Standard library comes with a module called sys that provides access to the 
interpreters, internals (a set of variables and functions available at runtime). 

One such function is exc_inf o, which provides information on the exception 
currently being handled. When invoked, exc_inf o returas a three-valued 
tuple where the first value indicates the exception’s type, the second details the 
exceptioni value, and the third contains a traceback object that provides 
access to the traceback message (should you need it). When there is no currently 
available exception, exc_inf o returas the Python null value for each of the 
tuple values, which looks like this: (None, None, None). 

Knowing all of this, leti experiment at the >>> shell. In the IDLE session that 
follows, weVe written some code thati always going to fail (as dividing by zero is 
never a good idea). A catch-all except suite uses the sys . exc_inf o function to 
extract and display data relating to the currently firing exception: 


To learn more 
ahout "sys ", see 
littos:/ / rfocs. 
pjtkon.org/ 3/ 
likrarj/ sjs.ktml. 


Python 3.5.2 Shell 


»> 


RESTART: Shell 


»> 

»> import sys ^r 
»> 

»> try: 

1/0 ^- 

except: 


err = sys.exc_info() 
for e in err: 


print(e) <£_ 


Be sure -to import the u sys” module- 

Dividing by z^ro is -^never-^ a good idea -and when your tode 
divides by ;zero an exception oddurs 

-Let's extradt and disylay the data assotiatcd 

_^ with the Currently oddurring exception- 




<class 'ZeroDivisionError'> 
division by zero 
<traceback object at 0xl05b22188> \ 

»> I 


Heve's the data assoeiated with the 
exception, whidh don-furws that we 
have an issue with divide-by-zevo. 

Ln: 1 17 Coi: 4 


Iti possible to delve deeper into the traceback object to learn more about what 
just happened, but this already feels like too much work, doesn’t it? All we really 
want to know is what type of exception occurred. 

To make this (and your life) easier, Python extends the try/except syntax to 
make it convenient to get at the information returned by the sys.exc info 
function, and it does this without you having to remember to import the sys 
module, or wrangle with the tuple returned by that function. 

Recall from a few pages back that the interpreter arranges exceptions in a 
hierarchy, with each exception inheriting from one called Exception. Let’s take 
advantage of this hierarchical arrangement as we rewrite our catch-all exception 
handler. 



Exception 
+ — 
+— 
+— 


+ - 
+ - 
+ - 
+ - 


Rc£3 II -fche 
exdep-fcion hicrardhy 

-Prom earlier. 


StopIteration 
StopAsyncIteration 
ArithmeticError 
+-- FloatingPointError 
+-- OverflowError 
+-- ZeroDivisionError 
AssertionError 
AttributeError 
BufferError 
EOFError 
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exception handling 


The Catch-AII Exception Handler, Revisited 

Gonsider your current code, which explicitly identifies the two exceptions you 
want to handle (FileNotFoundError and PermissionError), as well as 
provides a generic catch-all except suite (to handle everything else): 



# # # try examples4.py - /Users/paul/Desktop/NewBook/ch11/tr... 

TWis toAc 'N&rVs, Wt 

doesnt v-eally teli ^ -- 

you muth whcn SOimC 
unexfedted e*£epbon 

try : 

with open ( ’myfile.txt’ ) as fh: 

file data = fh.read() 
print(file data) 
except FileNotFoundError: 

print('The data file is missing.') 
except PermissionError: 

print('This is not allowed.') 
except: 

print('Some other error occurred. 1 ) 

odduvs. 

Ln: 12 Coi: 0 


Note how, when referring to a specific exception, weVe identified the exception by 
name after the except keyword. As well as identifying specific exceptions after 
except, it’s also possible to identify classes of exceptions using any of the names 
in the hierarchy. 

For instance, if you’re only interested in knowing that an arithmetic error 
has occurred (as opposed to—specifically—a divide-by-zero error), you 
could specify except ArithmeticError, which would then catch a 
FloatingPointError, an OverflowError, and a ZeroDivisionError 
should they occur. Similarly, if you specify except Exception, youil catch 
any error. 

But how does this help...surely you’re already catching all errors with a “bare” 
except statement? It’s true: you are. But you can extend the except 
Exception statement with the as keyword, which allows you to assign the 
current exception object to a variable (with err being a very popular name in 
this situation) and create more informative error message. Take a look at another 
version of the code, which uses except Exception as: 



Retall -fchdi all -(Jie 

exdep-fciohs ihhev-i-fc 
tvorrv “Exfiep-ti 


A OY\ 


Exception 

+-- Stoplteration 
+-- StopAsyncIteration 
+-- ArithmeticError 
| +-- FloatingPointError 

| +-- OverflowError 

| +-- ZeroDivisionError 

+-- AssertionError 
+-- AttributeError 
+-- BufferError 
+-- EOFError 















one last spin 



With this—the last of the changes to your try/except code—applied, let’s confirm that everything 
is working as expected before returning to vsearch4web. py and applying what you now know 
about exceptions to your webapp. 

Let’s start with confirming that the code displays the correct message when the file is missing: 


• • • Python 3.5.2 Shell 


»> 


- RESTART: /Users/paul/Desktop/ NewBook/chll/try examples5.py - 


The data file is missing. 


»> 


Ln: 7 Coi: 4 


"**y-Pil cixi" 

doesn'{ exisi 


If the file exists, but you don’t have permission to access it, a different exception is raised: 


• • Python 3.5.2 Shell 

»> 

======== RESTART: /Users/paul/Desktop/_NewBook/chll/try_examples5.py ======== 

The data file is missing. 

»> 

======== RESTART: /Users/paul/Desktop/_NewBook/chl1/try_examples5.py =======: 

This is not allowed. 

»> | 

Ln: 10 Coi: 4 


The file 
e*»sts> V>u*t 
you t 
v-ead »t 


Any other exception is handled by the catch-all, which displays a friendly message: 

• • • Python 3.5.2 Shell 

»> 

======== RESTART: /Users/paul/Desktop/_NewBook/chll/try_examples5.py ======== 

The data file is missing. 

»> 

======== RESTART: /Users/paul/Desktop/_NewBook/chl1/try_examples5.py ======== 

This is not allowed. 

»> 

======== RESTART: /Users/paul/Desktop/_NewBook/chll/try_examples5.py ======== 

Some other error occurred: [Errno 21] Is a directory: 'myfile.txt' 

»> 

Ln: 23 Coi: 4 


o-fcher 

exeeptioh has 

oeeuvvcd. this 
£ase, what you 
thought was a file 
is ih iaci a -Pol der. 



Finally, if all is OK, the try suite runs without error, and the file’s contents appear on screen: 


• • • Python 3.5.2 Shell 

»> 

======== RESTART: /Users/paul/Desktop/_NewBook/chl1/try_examples5.py ======== 

The data file is missing. 

»> 

======== RESTART: /Users/paul/Desktop/_NewBook/chll/try_examples5.py ======== 

This is not allowed. 

>» 

======== RESTART: /Users/paul/Desktop/_NewBook/chll/try_examples5.py ======== 

Some other error occurred: [Errno 21] Is a directory: 'myfile.txt' 

»> 

======== RESTART: /Users/paul/Desktop/_NewBook/chll/try_examples5.py ======== 

Empty (well... except for this line). 

>» 

Ln: 27 Coi: 4 


£uddcss| No 
e%deptio*s 
odduv-, so the 
U tv/ SUlte 'CUfhS 
to dompletioyt. 
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exception handling 


(retting Pack to Our Webapp Code 

Recall from the start of this chapter that we identified an issue with the call to 
log_request within vsearch4web . py’s do search function. Specifically, 
we’re concerned about what to do when the call tolog request fails: 


@app.route(’/search4 ', methods=['POST']) 
def do_search() -> ’html’: 

phrase = request.form[’phrase'] 

letters = request.form[’letters’] 

title = 'Here are your results:' 

r e ° ^ - ol-l \aeQidd- T i,LLc- s bhrase, letters) ) 

log_request (request , results) _4"* Whai happc^s 

returrr*^ d-? ~±- ^™p,1 L Vpe c,1 J-ht^'Thtml ’ , i-f this tali -fails? 

the_title=title, 
the_phrase=phrase, 
the_letters=letters , 
the results=results,) 


Based on our investigations, we learned that this call might fail if the backend 
database is unavailable, or if some other error occurs. When an error (of any 
type) occurs, the webapp responds with an unfriendly error page, which is likely to 
confuse (rather than enlighten) your webapp’s users: 



■ mysql.connector.errors.lntc 





c 


Q 127.0.0.1:5000/search4 


☆ 


mysql.connector.errors.InterfaceError 



This is nof. 
somcthin^ Y ou 
viant Y<xrt- 
weba\T usevs 
{a see- 


mysql.connector.errors.InterfaceError: 2003: Can’t connect to MySQL server on '127.0.0.1:3306' (61 
Connection refused) 


Traceback (most recent call last) 


File 7Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/flask/app.py", line 
2000, in_call_ 

return self.wsgi_app(environ, start_response) 

File 7Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/flask/app.py", line 
1991, in wsgi_app 

response = self.make_response(self.handle_exception(e)) 

File '7Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/flask/app.py", line 


Although it is important to us, the logging of each web request is not something 
that our webapp users really care about; all they want to see is the results of their 
search. Gonsequently, let’s adjust the webapp’s code so that it deals with errors 
within log_request by handling any raised exceptions silently. 
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don’t make a sound 


Silently Handling Exceptions 



Seriously? YouVe planning to handle 
exceptions raised by "log_request" silently? 
Isn't that just another variant of ignoring 
exceptions and hoping they'11 go away? 


No: “silently” does not mean “ignore.” 

When we suggest handling exceptions silently in this 
context, we’re referring to handling any exceptions 
raised in such a way that your webapp users don’t 
notice. At the moment, your users do notice, as 
the webapp crashes with a confusing and—let’s be 
honest —scary error page. 

Your webapp users don’t need to worry about 
log_request failing, but you do. So let’s adjust 
your code so that exceptions raised by log_ 
reque st aren’t noticed by your users (i.e., they 
are silenced), but are noticed by you. 


there icire nQ 

Dumb Questfens 



Doesrft all this try/except stuff just make my code harderto read and understand? 


Ifs true that the example code in this chapter started out as three easy-to-understand lines of Python code, and then we added seven 
lines of code, which—on the face of things—have nothing to do with what the first three lines of code are doing. However, it is important to 
protect code that can potentially raise an exception, and try/except is generally regarded as the best way to do this. Over time, your 
brain will learn to spot the important stuff (the code actually doing the work) that lives in the try suite, and filter out the except suites that 
are there to handle exceptions. When trying to understand code that uses try/except, always read the try suite first to learn what the 
code does, then look at the except suites if you need to understand what happens when things go wrong. 
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exception handling 



Let's add some try/except code to do_search's invocation of the 
log_request function.To keep things straightforward, let's add a catch- 
all exception handler around the call to log_request, which, when it 
fires, displays a helpful message on Standard output (using a call to the 
print BIF). In defining a catch-all exception handler, you can suppress 
your webapp's Standard exception-handling behavior, which currently 
displays the unfriendly error page. 

Here's log request's code as it's currently written: 


@app.route(’/search4', methods=['POST']) 
def do_search() -> 'html': 

phrase = request.form['phrase'] 
letters = request.form['letters'] 
title = 'Here are your results:' 
results = str (search41etters(phrase, letters)) 

log_request(request, results) ^- 

return render_template('results.html', 

the_title=title, 
the_phrase=phrase, 
the_letters=letters, 
the results=results,) 


This Ime o-f code needs 
io te piro-tec-ted 'm 
t-Sse it -fails (vaisiha a 
v-uivtir«e error). 


In the spaces below, provide the code that implements a catch-all 
exception handler around the call to log request: 


@app.route ( '/search4', methods=['POST']) 
def do_search() -> 'html': 

phrase = request.form['phrase'] 

letters = request.form['letters'] 

title = 'Here are your results:' 

results = str (search41etters(phrase, letters)) 


porU h^e-t -to tali 
"|oajre<\<Aest ? a ' r ^ 
oftte tode 70« add- 



return render_template('results.html', 

the_title=title, 
the_phrase=phrase, 
the_letters=letters, 
the results=results,) 
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catch it all 



The plan was to add some try/except code to do_search's 
invocation of the log_request function.To keep things 
straightforward, we decided to add a catch-all exception handler around 
the call to log_request, which, when it fires, displays a helpful 
message on Standard output (using a call to the print BIF). 

Here's log request's code as currently written: 


@app.route('/search4', methods=['POST’]) 
def do_search() -> 'html': 

phrase = request.form['phrase'] 

letters = request.form['letters' ] 

title = 'Here are your results:' 

results = str (search4letters(phrase, letters)) 

log_request(request, results) 

return render_template(’results.html’, 

the_title=title, 
the_phrase=phrase, 
the_letters=letters, 
the results=results,) 


In the spaces below, you were to provide the code that implements a 
catch-all exception handler around the call to log request: 


@app.route(’/search4’, methods=[’POST’]) 
def do_search() -> 'html': 

phrase = request.form['phrase'] 

letters = request.form['letters'] 

title = 'Here are your results:' 

results = str(search41etters(phrase, letters)) 


TW»s is “the 

taUh-all 




tv-y: 

results) 

tior\ as cv\r* 

L-°39 m 3 -failcd v/ith this errori s*br(err)) <7 


The Cal1 io "log_jre^uest" i s 
m r L ed Jthe sui ie associa-ted 

Wl ^ h a hew try" siaie*e»t 


return render_template('results.html', 

the_title=title, 
the_phrase=phrase, 
the_letters=letters, 
the results=results,) 


VVV>ev> a vwfcwe error ottuvs, this 
mcssayt is d.sylayed stree» W 
■the adr»m only- Y<wv user sees 
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exception handling 



With the catch-all exception-handling code added to vsearch4web. py, let’s take your webapp for an 
extended spin (over the next few pages) to see the difference this new code makes. Previously, when 
something went wrong, your user was greeted with an unfriendly error page. Now, however, the error is 
handled “silently” by the catch-all code. If you haven’t done so already, run vsearch4web. py, then 
use any browser to surf to your webapp’s horne page: 


$ python3 vsearch4web.py 

* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) 

* Restarting with fsevents reloader 

* Debugger is active! 

* Debugger pin code: 184-855-980 


' , '- va Xx '5 U p and ruKmiha, 

" a * h 5 a browser... 



6fo ahead 
and suv-f oy \ 
ovev- to youv 
weba^s home 

?ay- 



Welcome to search4letters on the web! 

Use this forrri to submit a search request: 


Phrase: i 
Letters: aeiou 

When you're ready, click this button: 

Do it! 


On the terminal that’s running your code, you should see something like this: 



* Debugger pin code: 184-855-980 

127.0.0.1 - - [14/Jul/2016 10:54:31] 

127.0.0.1 - - [14/Jul/2016 10:54:31] 

127.0.0.1 - - [14/Jul/2016 10:54:32] 

Tbese 2-OOs tcm-firm thal your weka^ 
is UP and (ard senmj ■*? ** 

V,ome ? ay). A" « * iW,s ? omt ' 


'GET / HTTP/1.1" 200 - 
'GET /static/hf.css HTTP/1.1" 200 - 
'GET /favicon.ico HTTP/1.1" 404 - 



?J ^ ibis W...we haver.1 

(!*' d » Wito' -Pile 4r our webapp 

/° ' l S^pP^-ted as ho-t Pound when your 
browser asks +or ii). 7 


you are here i 
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test test test 



In order to simulate an error, we’ve switched off our backend database, which should resuit in an error 
occurring whenever the webapp tries to interact with the database. As our code silently catches all errors 
generated by log_request, the webapp user isn’t aware that the logging hasn’t occurred. The catch- 
all code has arranged to generate a message on screen describing the problem. Sure enough, when 
you enter a phrase and click on the “Do it!” button, the webapp displays the results of your search in the 
browser, whereas the webapp’s terminal screen displays the “silenced” error message. Note that, despite 
the runtime error, the webapp continues to execute and successfully Services the call to /search: 


••• 

0 Welcome to search4letter X 

• 

* 

o 

I D 127 . 0 . 0 . 1:5000 

&\ = 


Welcome to search4letters on the web! 

Use this form to submit a search request: 


Phrase: j Testing out catch all code 


Letters: 


testing 


When you're ready, click this button: 


Do it! 



Here are your results: 

You submitted the following data: 


Phrase: 

Testing out catch all code 

Letters: 

testing 


When "Testing out catch all code" is search for "testing", the following results are 
returned: 

{'e', 'n', 'g', 's', 't', 'i'} 



127.0.0.1 - - [14/Jul/2016 10:54:32] "GET /favicon.ico HTTP/1.1" 404 - 

***** Logging failed with this error: 2003: Can't connect to MySQL server on '127.0.0.1:3306' 
(61 Connection refused) 

127.0.0.1 - - [14/Jul/2016 10:55:55] "POST /search4 HTTP/1.1" 200 - 


This message is J»erated \>) your taUb-a« 
e*tepWhandH tode TV,e ' Neba?? ^ 
docsy\ “t scc it 



t e l 3h *** oU ^ e <t’ -the webapp didn't 
asb. In other words, the seardh worked 0>ut the 

webapp User ,sn i aware that the logging failed) 
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exception handling 



(Extsiwqd) Tqst DriI/q 


q 3 9P 3 

In fact, no matter what error occurs when log_request runs, the catch-all code handles it. 

We restarted our backend database, then tried to connect with an incorrect username. You can raise this 
error by changing the dbconf ig dictionary in vsearch4web. py to use vsearchwrong as the 
value for user: 


app.config['dbconfig'] = {'host': '127.0.0.1’, 

'user': 'vsearchwrong', - 

'password': 'vsearchpasswd', 

'database': 'vsearchlogDB', } 


When your webapp reloads and you perform a search, you’11 see a message like this in your terminal: 

& 

***** Logging failed with this error: 1045 (28000): Access denied for user 'vsearchwrong'@ 

'localhost' (using password: YES) 









Change the value for user back to vsearch, and then let’s try to access a nonexistent table, 
by changing the name of the table in the SQL query used in the log_request function to be 
logwrong (instead of log): 


def log request(req: 'flask_request', res: str) -> None 
with UseDatabase(app.config['dbconfig']) as cursor: 
SQL = """insert into logwrong 


(phrase, letters, ip, browser string, results) 
values 

/ o, 0 o. g, o, 0 9. 0 \ ii ii ii 

i ob, o S , oS, oS, oSy 


* * * * * 


When your webapp reloads and you perform a search, you’11 see a message like this in your terminal: 


Logging failed with this error: 1146 (42S02): Table 'vsearchlogdb.logwrong' doesn't exist 



Change the name of the table back to log and then, as a final example, let’s add a raise statement to 
the log request function (just before the with statement), which generates a custom exception: 


def log request(req: 'flask_request', res: str) -> None: 
raise Exception("Something awful just happened.") 
with UseDatabase(app.config['dbconfig']) as cursor: 


* * * * * 


When your webapp reloads one last time, and you perform one last search, you’ll see the following 
message in your terminal: 


Logging failed with this error: Something awful just happened 
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handling more errors 


Handling Other Patabase Errors 


The log_request function makes use of the UseDatabase context manager (as 
provided by the DBcm module). Now that youVe protected the call to log_request, 
you can rest easy, safe in the knowledge that any issues relating to problems with the 
database will be caught (and handled) by your catch-all exception-handling code. 


However, the log_request function isn’t the only place where your webapp 
interacts with the database. The view_the_log function grabs the logging data 
from the database prior to displaying it on screen. 

Recall the code for the view the log function: 


AII of “this Code 

needs -to be 

P roletled, loo. 



@app.route('/viewlog') 

@check_logged_in 
def view_the_log() -> 'html': 

with UseDatabase(app.config['dbconfig']) as cursor: 

_SQL = """select phrase, letters, ip, browser_string, results 

from log""" 
cursor.execute(_SQL) 
contents = cursor.fetchall () 
tities = ( 1 Phrase 1 , 'Letters', 'Remote_addr ', 

return render_template('viewlog.html', 

the_title= 1 View Log' , 
the_row_titles=titles , 
the data=contents,) 


'User agent', 'Results') 


This code can fail, too, as it interacts with the backend database. However, unlike 
log_request, the view_the_log function is not called from the code in 
vsearch4web . py; ifs invoked by Flask on your behalf. This means you can’t write 
code to protect the invocation of view_the_log, as ifs the Flask framework that 
calls the function, not you. 

If you can’t protect the invocation of view_the_log, the next best thing is to 
protect the code in its suite, specifically the use of the UseDatabase context 
manager. Before considering how to do this, lef s consider what can go wrong: 


The backend database may be unavailable. 

You may not be able to log in to a working database. 
After a successful login, your database query might fail. 
Something else (unexpected) might happen. 


This list of problems is similar to those you had to worry about with log_request. 
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Pocs "More Errors" Meaw "More excepts"? 

Knowing what we now know about try/except, we could add some code to the 
view_the_log function to protect the use of the UseDatabase context manager: 


Anothev- 

eatfch-all 

ereptio* 

ha^dlev- 


@app.route( 1 /viewlog *) 

@check_logged_in 
def view_the_log() -> 1 html’: 

try: 

with UseDatabase(app.config[ ! dbconfig 1 ]) 



except Exception as err: 

print('Something went wrong: ', str (err) ) 


as 


cursor: 

^ — 


The \rest o-f 
the -Puhetioh^s 
£ode goes 
he\re. 



This catch-all strategy certainly works (after ali, that 5 s what you used with log_ 
reque st). However, things can get complicated if you decide to do something other 
than implement a catch-all exception handler. What if you decide you need to react 
to a specific database error, such as “Database not found”? Recall from the beginning 
of this chapter that MySQL reports an InterfaceError exception when this 
happens: 


| mysql.connector.errors.lnt X 


C □ 127.0.0.1:5000/search4 


☆ 


mysql.connector.errors.InterfaceError 

mysql.connector.errors.InterfaceError: 2003: Can't connect to MySQL server ori '127.0.0.1:3306' (61 
Cormection refused) 


Traceback (most recent call last) 


File "/Library/Frameworks/FVthon.framework/Versions/B.S/lib/pythonB.S/site-packages/flask/app.py", line 2000 , 
in_call_ 

return self.wsgi_app(environ, start_response) 

File "/Library/Frameworks/FVthon.framework/Versions/B.S/lib/pythonB.S/site-packages/flask/app.py", line 1991 , 
in wsgi_app 

response = self.make_response(self.handle_exception(e)) 

File 'VLibrary/Frameworks/FVthon.framework/Versions/B.S/lib/pythonB.S/site-packages/flask/app.py", line 1567 , 
in handle_exception 


This t%tt ption 
is v-aised wheh 
youir todt 
teni -fihd 
the baekehd 
database. 


You could add an except statement that targets the InterfaceError exception, 
but to do this your code also has to import the mysql .connector module, which 
defines this particular exception. 

On the face of things, this doesn’t seem like a big deal. But it is. 
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watch your imports 


Avoid Tightly Coupled Code 

Lefs assume youVe decided to create an except statement that protects against your 
backend database being unavailable. You could adjust the code in view_the_log to 
look something like this: 


@app.route(’/viewlog f ) 

@check_logged_in The v-esi o-f 

def view_the_log () -> ' html' : -fche s 

try: tode stili goes 

with UseDatabase(app.config['dbconfig']) as cursor: i» heve. 

^_ J 


except mysql.connector.errors.InterfaceError as 
print('Is your database switched on? Error:’ 
except Exception as err: 

print('Something went wrong:’, str (err) ) 


If you also remember to add import mysql. connector to the top of your code, 
this additional except statement works. When your backend database can’t be found, 
this additional code allows your webapp to remind you to check that your database is 
switched on. 


err: 

f str(err)) 



j\Ad anothev 
statement to handle 

a s ? e<Uit e*teptio« 


This new code works, and you can see whafs going on here... whafs not to like? 

The issue with approaching the problem in this way is that the code in 
vsearch4web . py is now very tightly coupled to the MySQL database, and 
specifically the use of the MySQL Connector module. Prior to adding this second 
except statement, your vsearch4web . py code interacted with your backend 
database via the DBcm module (developed earlier in this book). Specifically, the 
UseDatabase context manager provides a convenient abstraction that decouples 
the code in vsearch4web . py from the backend database. If, at some time in the 
future, you needed to replace MySQL with PostgreSQL , the only changes you’d need 
to make would be to the DBcm module, not to all the code that uses UseDatabase. 
However, when you create code like that shown above, you tightly bind (i.e., couple) 
your webapp code to the MySQL backend database because of that import 
mysql .connector statement, in addition to your new except statements 
reference to mysql. connector . errors . Interf aceError. 



If you need to write code that tightly couples to your backend database, always 
consider putting that code in the DBcm module. This way, your webapp can be written 
to use the generic interface provided by DBcm, as opposed to a specific interface that 
targets (and locks you into) a specific backend database. 

Lefs now consider what moving the above except code into DBcm does for our 
webapp. 
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The PPcm Module, Revisited 

You last looked at DBcm in Chapter 9, when you created that module in order to 
provide a hook into the with statement when working with a MySQL database. Back 
then, we sidestepped any discussion of error handling (by conveniently ignoring the 
issue). Now that you Ve seen what the sys . exc_inf o function does, you should have 
a better idea of what the arguments to UseDatabase’s_exit_method mean: 

import mysql.connector 
class UseDatabase: 

def init (self, config: dict) -> None: 

self.configuration = config 

def _enter_(self) -> 'cursor': 

self.conn = mysql.connector.connect(**self.configuration) 
self.cursor = self.conn.cursor() 
return self.cursor 

def exit (self, exc_type, exc_value, exc_trace) -> None: 

self.conn.commit() 
self.cursor.close() 
self.conn.close() 

Recall that UseDatabase implements three methods: 

• init provides a configuration opportuni typrior to with executing, 

• _enter_executes as the with statement starts, and 

• _ exit _is guaranteed to execute whenever the with 5 s suite terminates. 


L No\u -fcha-fc youve 
•i should be tlcar 

-fchcse *c£hod 

^^gumch-ts ire-Per 

cx^cpiioh da-fca. 


This is “the 
**ana$ev- todt 

m W P 



At least, that’s the expected behavior whenever everything goes to plan. When things 
go wrong, this behavior changes. 

For instance, if an exception is raised while_ enter _is executing, the with 

statement terminates, and any subsequent processing of_ exit _is cancelled. This 

makes sense: if_ enter _ runs into trouble,_ exit _can no longer assume that 

the exeeution context is initialized and configured correctly (so it’s prudent not to run 
the_ exit _ method 5 s code). 

The big issue with the_ enter _methodk code is that the backend database may 

not be available, so let’s take some time to adjust_ enter _for this possibility, 

generating a custom exception when the database connection cannot be established. 
Once weVe done this, well adjust view_the_log to check for our custom 
exception instead of the highly database-specific mysql. connector. errors. 
Inter f aceError. 
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Creating Custom Exceptions 

Greating your own custom exceptions coukkft be any easier: decide on an appropriate 
name, then define an empty class that inherits from Python’s built-in Exception 
class. Once youVe defined a custom exception, it can be raised with the raise 
keyword. And once an exception is raised, it 5 s caught (and dealt with) by try/ 
except. 

A quick trip to IDLE’s >>> prompt demonstrates custom exceptions in action. In this 
example, we’re creating a custom exception called ConnectionError, which we 
then raise (with raise), before catching with try/except. Read the annotations in 
numbered order, and (if you’re following along) enter the code weVe typed at the >>> 
prompt: 


|. Cv-eate a 
new dlass tallcd ^ 

"Comtthorfcrror" 

■that 'mhev'rb tv-om 
the "E*tepW' 
tlass. 



Python 3.5.2 Shell 


Ca^tdh the 
CXdeptioh usihA 


»> class ConnectionError(Exception) 
pass 


: s p y^° h ' s 

that crates the e«.pty elass. 


»> 

»> 

»> raise ConnectionError ( ’Cannot connect... is it time to panic?') 
Traceback (most recent call last): 

File "<pyshell#74> M , line 1, in <module> 

raise ConnectionError('Cannot connect... is it time to panic?') 
ConnectionError: Cannot connect... is it time to panic? 

»> 

>>> try: i 

raise ConnectionError (' Whoops!' ) 

except ConnectionError as err: £ W ) vesults 

print ( ' Got: ' , str (err) ) i . a 

mc ssa^c* 

Got: Whoops! 

»> 

»> I 





Ln: 148 Coi: 4 


5. The "ConnettionErv-or'’ ^ dau^ht, allov/m^ 


I . -L- 






The empty class isn't quite empty... 

In describing the ConnectionError class as “empty,” we told a little lie. 

Granted, the use of pass ensures that there 5 s no new code associated with the 
ConnectionError class, but the fact that ConnectionError inherits from 
Python 5 s built-in Exception class means that all of the attributes and behaviors 
of Exception are available in ConnectionError too (making it anything but 
empty). This explains why ConnectionError works just as you’d expect it to with 
raise and try/except. 
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Lefs adjust the DBcm module to raise a custom ConnectionError 
whenever a connection to the backend database fails. 

Here's the current code to DBcm. py. In the spaces provided, add in the 
code required to raise a ConnectionError. 


pc-f mc youv 
dus-to** e^deftiorv 


import mysql.connector 



class UseDatabase: 


def _init_(self, config: dict) -> None 

self.configuration = config 


def 


enter (self) -> 'cursor': 


self.conn = mysql.connector.connect(**self.configuration) 
self.cursor = self.conn.cursor () 
return self.cursor 


Add £ ode 

to 'Va.se" a 

"Comncthonf-^or" 



def 


exit (self, exc type, exc value, exc trace) -> None 


Use youv pe*d»l -to 
show ‘the dhav^cs 
you d **ake “to “this 
dode v\ow “that “the 
TowedWfcvvov” 

e^deftio^ elisis. 




self.conn.commit() 
self.cursor.close() 
self.conn.close() 

With the code in the DBcm module amended, use your pencil to detail any 
o changes you'd make to this code from vsearch4web . py in order to 
take advantage of the newly defined ConnectionError exception: 


from DBcm import UseDatabase 
import mysql.connector 


the_row_titles=titles, 
the_data=contents,) 

except mysql.connector.errors.InterfaceError as err: 

print('Is your database switched on? Error:', str(err)) 
except Exception as err: 

print('Something went wrong:', str(err)) 
return 'Error' 
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raising a connectionerror 

%Jharpen your pencil 

Solution 




O 


non 


import mysql.connector 


Penine the 
dus-to** exdept 
as an w er»pt/ 
dlass that 
mhevrls 4Voro 

u E*dept»on . dlass ConnedtionEv-v-oir(Exdeption): 
^ ^ pass 

class UseDatabase: 


You were to adjust the DBcm module to raise a custom 
ConnectionError whenever a connection to the backend database 
fails. You were to adjust the current code to DBcm. py to add in the code 
required to raise a ConnectionError. 


def 


init (self, config: dict) -> None: 


def 


self.configuration = config 
enter (self) -> 'cursor': 


new W *bry/ 
e^dept” 
donstvudt 
pv-otedts the — 
database 

donnCdtion dode- 


bcy 


WitWm the "D&tm?/ toAt ' 
vc^ev* to the badkcnd 
database-s^eti^it e*tepbons 
bv tbeiv tll na»»e- 


self.conn = mysql.connector.connect(**self.configuration) 
self.cursor = self.conn.cursor() 
return self.cursor 




exdept mys^Uonnedto\r e\r\ro\rs.|ntev-'fadeE\r\ro\r as evnr: 
vaise Connedtion&rirovtevr) - 


Raise the e usto 
exdeption. 


def 


exit (self, exc type, exc value, exc trace) -> None: 


self.conn.commit() 
self.cursor.close() 
self.conn.close() 


ybu dont need 
to iwpovt 
w r*ys<\l Connefitov- 
anymov-e (a* 
w pDdw” does 

this -£ov You) • ^ 

from DBcm import UseDatabase, ConnedxionfcVYoV 


e 



With the code in the DBcm module amended, you were to detail any 
changes you'd make to this code from vsearch4web . py in order to 
take advantage of the newly defined ConnectionError exceptioni 


Be su\re to 
impoirt the 

ConnedtionEvror” 
exdeption -firono 
u PBd* w 


Change the tivst 
u exdept" statement 

to look to\r a 

ConnedtionEWov” 
as opposed to an 
u |ntertade£vror" 


Connedtion&rvov 


except - 


the_row_titles=titles, 
the_data=contents, ) 

Terrol^s-r 


as err: 


print(’Is your database switched on? Error:’, str(err)) 
except Exception as err: 

print(’Something went wrong:’, str(err)) 
return 'Error' 
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Tesr DriVs- 

Let’s see what difference this new code makes. Recall that you’ve moved the MySQL-specific 
exception-handling code from vsearch4web. py into DBcm. py (and replaced it with code that 
looks for your custom ConnectionError exception). Has this made any difference? 

Here are the messages that the previous version of vsearch4web. py generated whenever the 
backend database couldn’t be found: 

Is your database switched on? Error: 2003: Can't connect to MySQL server on '127.0.0.1:3306' 
(61 Connection refused) 

127.0.0.1 - - [16/Jul/2016 21:21:51] "GET /viewlog HTTP/1.1" 200 - 

And here are the messages that the most recent version of vsearch4web. py generates whenever 
the backend database can’t be found: 


Is your database switched on? Error: 2003: Can't connect to MySQL server on '127.0.0.1:3306' 
(61 Connection refused) 

127.0.0.1 - - [16/Jul/2016 21:22:58] "GET /viewlog HTTP/1.1" 200 - 



o 


YouVe trying to 
trick me, aren't you? 
These error messages 
are the same! 


Yes. On the face of things, these are the same. 

However, although the output from the current and previous 
versions of vsearch4web . py appears identical, behind the 
scenes things are very different. 

If you decide to change the backend database from MySQL to 
PostgreSQL , you no longer have to worry about changing any 
of the code in vsearch4web . py, as all of your database- 
specific code resides in DBcm . py. As long as the changes you 
make to DBcm . py maintain the same interjace as previous 
versions of the module, you can change SQL databases as 
often as you like. This may not seem like a big deal now, but if 
vsearch4web . py grows to hundreds, thousands, or tens of 
thousands of lines of code, its really is a big deal. 
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What Else Can <ro Wrong with "PPcw"? 

Even if your backend database is up and running, things can stili go wrong. 

For example, the credentials used to access the database may be incorrect. If 

they are, the_ enter _method will fail again, this time with a mysql. 

connector.errors.ProgrammingError. 

Or, the suite of code associated with your UseDatabase context manager 
may raise an exception, as there 5 s never a guarantee that it executes correctly. A 
mysql. connector . errors . ProgrammingError is also raised whenever 
your database query (the SQL you’re executing) contains an error. 

The error message associated with an SQL query error is different than the 
message associated with the credentials error, but the exception raised is the same: 
mysql. connector . errors . ProgrammingError. Unlike with credentials 
errors, errors in your SQL results in an exception being raised while the with 
statement is executing. This means that youll need to consider protecting against 
this exception in more than one place. The question is: where? 

To answer this question, let’s take another look at DBcrrfs code: 


import mysql.connector 


class ConnectionError(Exception): 
pass 

class UseDatabase: 

def _init_(self, config: dict): 

self.configuration = config 


This Lo&t 

Ldn v-aise a w 

K pvo$vam*"'>"a£' r ' rov ' 
enteptio*- \ 


def 


enter (self) -> 'cursor': 


try: 



self.conn = mysql.connector.connect(**self.configuration) 
self.cursor = self.conn.cursor() 
return self.cursor 

except mysql.connector.errors.InterfaceError as err: 
raise ConnectionError(err) 

_exit_(self, exc_type, exc_value, exc_traceback): 

self.conn.commit() 
self.cursor.close() 
self.conn.close() 




But uibat about enteptions tbat Ott uv 
wtWm tbe V,tb" suite? Ttee ba ?? e» 
*atter* tbe K _eUer_ «etbod yds 

but *beW* tbe "_ ey.it — "<tbod 

s-tav-ts. 


You might be tempted to suggest that exceptions raised within the 
with suite should be handled with a try/except statement 
within the with, but such a strategy gets you right back to writing 
tightly coupled code. But consider this: when an exception is 
raised within with’s suite and not caught, the with statement 
arranges to pass details of the uncaught exception into your 

context managebs_ exit _method, where you have the 

option of doing something about it. 
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Creating More Custom Exceptions 

Let 5 s extend DBcm . py to report two additional, custom exceptions. 

The first is called CredentialsError and is raised when a ProgrammingError 

occurs within the_ enter _method. The second is called SQLError and is raised 

when a ProgrammingError is reported to the_ exit _method. 

Defining these new exceptions is easy: add two new, empty exception classes to the top 
of DBcm . py: 

import mysql.connector 

class ConnectionError(Exception): 
pass 

class CredentialsError(Exception): 
pass 

class SQLError(Exception): 
pass 

class UseDatabase: 

def _init_(self, configurationi dict): 

self.config = configuration 



Tv/o addrfciohal 
classes dc-fihc youv 
■two hCW cxdcp-ti 


iohs. 


A CredentialsError can occur during_ enter _, so let’s adjust that method’s 

code to reflect this. Recall that an incorrect MySQL username or password results in a 
ProgrammingError being raised: 


Add tWis todc p 

i\\c "_enter_ " 

method L deal 

any lo^in ' isswes ' 



try: 

self.conn = mysql.connector.connect(**self.config) 
self.cursor = self.conn.cursor () 
return self.cursor 

except mysql.connector.errors.InterfaceError as err: 
raise ConnectionError(err) 

except mysql.connector.errors.ProgrammingError as err: 
raise CredentialsError(err) 


def _exit_(self, exc_type, exc_value, exc_traceback): 

self.conn.commit() 
self.cursor.close() 
self.conn.close() 


These code changes adjust DBcm.py to raise a CredentialsError exception 
when you provide either an incorrect username or password from your code to your 
backend database (MySQL). Adjusting vsearch4web . py 5 s code is your next task. 
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Are Your Patabase Credewtials Correct? 


With these latest changes made to DBcm. py, let’s now adjust the code in 
vsearch4web . py, paying particular attention to the view_the_log function. 
However, before doing anything else, add CredentialsErrorto the list of 
imports from DBcm at the top of your vsearch4web . py code: 


Be suve -fco 
inr»pov~t youv 
hCw cx^cp-tioh. 


from DBcm import UseDatabase, 


ConnectionError, CredentialsError 



With the import line amended, you next need to add a new except 
suite to the view_the_log function. As when you added support for a 
ConnectionError, this is a straightforward edit: 


@app.route('/viewlog') 
@check_logged_in 
def view_the_log() -> 'html': 
try: 


with UseDatabase(app.config['dbconfig']) 

_SQL = MMM select phrase, letters, ip, 
from log""" 
cursor.execute(_SQL) 
contents = cursor.fetchall() 
tities = ('Phrase', 'Letters', 'Remote_addr' , 
return render_template('viewlog.html', 

the_title='View Log', 
the_row_titles=titles, 
the_data=contents,) 
except ConnectionError as err: 

print('Is your database switched on? Error:', 
except CredentialsError as err: 

print('User-id/Password issues. Error:', 
except Exception as err: 

print('Something went wrong:', str(err)) 
return 'Error' 


as cursor: 
browser string. 


results 


'User agent', 'Results') 


r str(err)) 
str(err)) 



There’s really nothing new here, as all you’re doing is repeating what you did 
for ConnectionError. Sure enough, if you try to connect to your backend 
database with an incorrect username (or password), your webapp now displays an 
appropriate message, like this: 


ta-ta «he* your toAc uses 
■the '«vokOj usev*ame or password 

With MySQU 


User-id/Password issues. Error: 1045 (28000): Access denied for user 'vsearcherror'@'localhost' 
(using password: YES) 

127.0.0.1 - - [25/Jul/2016 16:29:37] M GET /viewlog HTTP/1.1" 200 - 

y°w iode khows all about 
Cv-edeM-tialsEv-ror', you generate an exeeption- 
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Handling SQlError 1$ Pifferent 

Both ConnectionError and CredentialsError are raised due to 

problems with the_enter_method’s code executing. When either exception 

is raised, the corresponding with statemenfs suite is not executed. 

If all is well, your with suite executes as normat 

Recall this with statement from the log_request function, which uses the 
UseDatabase context manager (provided by DBcm) to insert data into the 
backend database: 


with 

r 

Wc vced -to wov-v-y 
abou-t yibat ba??«» s 

*i£ some-lWmJ <y>C s ^_ i 

wvoyi^ wrtb “this 
toAt c > “tbc toAt 

WrlWm -tbe Vrlh” 
surbe)- v- 


UseDatabase(app.config['dbconfig' ]) as cursor: 

_SQL = """insert into log 

(phrase, letters, ip, browser_string, 
values 

/9-0 9-0 9-0 9-0 9- O \ HI!!! 

^ O O f OOf O uD f OO^ OO/ 

cursor.execute(_SQL, (req.form['phrase'], 

req.form[ 1 letters 1 ], 
req.remote_addr, 
req.user agent.browser, 
res, )) 


results) 


If (for some reason) your SQL query contains an error, the MySQL Connector 
module generates a ProgrammingError, just like the one raised duringyour 

context managebs_enter_method. However, as this exception occurs within 

your context manager (i.e., within the with statement) and is not caught there, the 

exception is passed back to the_exit_method as three arguments: the type 

of the exception, the value of the exception, and the traceback associated with the 
exception. 

If you take a quick look at DBcrrfs existing code for_exit_, you’11 see that the 

three arguments are ready and waiting to be used: 



Tbe -fchiree 
cx^p-tio h 

ave 

ready -for «se. 


def _exit_(self, exc_type, exc_value, exc_traceback): 

self.conn.commit() 
self.cursor.close() 
self.conn.close() 


When an exception is raised within the with suite and not caught, the context 

manager terminates the with suite’s code, and jumps to the_exit_ 

method, which then executes. Knowing this, you can write code that checks for 
exceptions of interest to your application. However, if no exception is raised, the 
three arguments (exc_type, exc_value, and exc_traceback) are all set to 
None. Otherwise, they are populated with details of the raised exception. 

Let 5 s exploit this behavior to raise an SQLError whenever something goes wrong 
within the UseDatabase context manager’s with suite. 


ttlT M • 

None is 
Pytnon’s 
null value. 
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exctype exc_value exc_traceback 


Pe Careful with Code Positiowiwg 


To check whether an uncaught exception has occurred within your code’s with 

statement, check the exc_type argument to the_exit_method within 

exit_’s suite, being careful to consider exactly where you add your new code. 




Yes, it does make a difference. 

To understand why, consider that your context 

manager’s_exit_method provides a place 

where you can put code that is guaranteed to 
execute after your with suite ends. That behavior is 
part of the context manager protocol, after ali. 

This behavior needs to hold even when exceptions 
are raised within your context manageks with 
suite. Which means that if you plan to add code 

to the_exit_method, it 5 s best to put it after 

any existing code in_exit_, as that way youil 

stili guarantee the method’s existing code executes 
(and preserve the semantics of the context manager 
protocol). 


Let’s take another look at the existing code in the_exit_method in light of 

this code placement discussion. Consider that any code we add needs to raise an 
SQLError exception if exc_type indicates a ProgrammingError has occurred: 


|jf you add tode m bev-e, 
and -tha-t io<k raiscs 
an e*fce\>t»on, “tbe -tVee 
e*»st*nj lifics o£ 6odc 
wont be e%e£u-tcd- 


def 


_exit_(self, exc_type, 

self.conn.commit() 
self.cursor.close () 
self.conn.close () 




exc value, exc traceback): 




* - Z — docs thino 

Jik a " y passed_ih exdepii ° h is deal ± 
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Raising ah SQLError 

At this stage, youVe already added the SQLError exception class to the top of the 
DBcm . py file: 


import mysql.connector 

class ConnectionError(Exception): 
pass 


class CredentialsError(Exception): 
pass 

class SQLError(Exception): 
pass 



WevVs whcv~c 
you added ih 

exception. 


class UseDatabase: 

def _init_(self, config: dict): 

self.configuration = config 


With the SQLError exception class defined, all you need to do now is add some 

code to the_exit_method to check whether exc_type is the exception you’re 

interested in, and if it is, raise an SQLError. This is so straightforward that 
we are resisting the usual Head First urge to turn creating the required code into an 
exercise, as no one wants to insuit anyone’s intelligence at this stage in this book. So, 
here’s the code you need to append to the_exit_method: 


1$ a ''PvoOjva™>*mO|£vv-ov 

ottwv-s, v-a<se a* 
"SQLEvv-o/- 


def _exit_(self, exc_type, exc_value, exc_traceback): 

self.conn.commit() 
self.cursor.close () 
self.conn.close () 

if exc type is mysql.connector.errors.ProgrammingError: 
raise SQLError(exc value) 



If you want to be extra safe, and do something sensible with any other, unexpected 

exception sent to_exit_, you can add an elif suite to the end of the 

_ exit _method that reraises the unexpected exception to the calling code: 


self.conn.close () 

if exc_type is mysql.connector.errors.ProgrammingError: 

raise SQLError(exc_value) 
elif exc_type: 

raise exc type(exc value) 




This "elit" vaises 
any othev exception 
■that mi^ht oCCwv. 
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no more programmingerror 



With support for the SQLError exception added to DBcm.py, add another except suite to your 
view the log function to catch any SQLErrors that occur: 


Add *tWts dode 
m-bo i\\t 

y/rlWm youv- ^ 
webapp- 



except ConnectionError as err: 

print ('Is your database switched on? Error:', str (err)) 
except CredentialsError as err: 

print('User-id/Password issues. Error:', str (err)) 
except SQLError as err: 

print ('Is your query correct? Error:', str (err)) 
except Exception as err: 

print('Something went wrong: ' , str(err)) 
return 'Error' 


Once you save vsearch4web. py, your webapp should reload and be ready for testing. If you try 
to execute an SQL query that contains errors, the exception is handled by the above code: 


Is your query correct? Error: 1146 (42S02): Table 'vsearchlogdb.logerror' doesn't exist 
127.0.0.1 - - [25/Jul/2016 21:38:25] "GET /viewlog HTTP/1.1" 200 - 



No more ^enevit Tvo3V3mmmg&rW' e*deptiow> Lor» 
MySQL Connedbov, as youv dustom exdeftiott-hawdlunj dode 

datdhes -these evvovs now 


Equally, if something unexpected happens, your webapp’s catch-all code kicks into gear, displaying 
an appropriate message: 


Something went wrong: Some unknown exception. 

127.0.0.1 - - [25/Jul/2016 21:43:14] "GET /viewlog HTTP/1.1" 


200 



l-P somcthihg unexped-ted 
happchs, youv Codt 
handles i-t. 


With exception-handling code added to your webapp, no matter what runtime error occurs, your 
webapp continues to function without displaying a scary or confusing error page to your users. 



And the really nice thing about 
this is that this code takes the generic 
"ProgrammingError" exception provided by 
the MySQL Connector module and turns 
it into two custom exceptions that have 
specific meaning for our webapp. 


Yes, it does. And this is very powerful 
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A ftuick Recap: Addiwg Robustness 

Let’s take a minute to remind ourselves of what we set out to do in this chapter. 
In attempting to make our webapp code more robust, we had to answer four 
questions relating to four identified issues. Let’s review each question and note 
how we did: 


You created a new exception called ConnectionError that is raised whenever 
your backend database can’t be found. You then used try/except to handle a 
ConnectionError were it to occur. 


It was a “happy accident,” but your choice of Flash plus Jinja2, together with Python’s 
DB-API specification, protects your webapp from the most notorious of web attacks. So, 
yes, your webapp is protected from some web attacks (but not all). 


We stili haven’t answered this question, other than to demonstrate what happens when 
your webapp takes 15 seconds to respond to a user request: your web user has to wait (or, 
more likely, your web user gets fed up waiting and leaves). 


You used try/except to protect the function call, which allowed you to control what 
the user of your webapp sees when something goes wrong. 


© What happens if a function call fails? 


( ) What happens if something takes a long time? 


© Is our webapp protected from web attacks? 


O What happens if the database connection fails? 


What happens if something takes a long time? 

When you did the initial exercise at the start of this chapter, this question resulted 
from our examination of the cursor . execute calls that occurred in the 
log_request and view_the_log functions. Although you Ve already worked 
with both of these functions in answering questions 1 and 4, above, youVe not 
done with them quite yet. 

Both log_request and view_the_log use the UseDatabase context 
manager to execute an SQL query. The log_request function writes the 
details of the submitted search to the backend database, whereas the view_ 
the_log function reads from the database. 

The question is: what do you do if this write or read takes a long time? 

Well, as with a lot of things in the programming world, it depends. 
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How to Peal with Wait? It Pepends... 

How you decide to deal with code that makes your users wait—either on a read, or 
on a write—can get complex. So we’re going to pause this discussion and defer a 
solution until the next, short chapter. 

In fact, the next chapter is so short that it doesn’t warrant its own chapter number 
(as you’11 see), but the material it presents is complex enough to justify splitting it 
off from this chaptehs main discussion, which presented Python 5 s try/except 
mechanism. So, let’s hang on for a bit before putting to rest issue 3: what happens if 
something takes a long time? 



Yes. The irony is not lost on us. 

We’re asking you to wait to learn how to 
handle “waits” in your code. 

But you Ve already learned a lot in this 
chapter, and we think it 5 s important to 
take a bit of time to let the try/except 
material sink into your brain. 

So, we’d like you to pause, and take a short 
break...after you Ve cast your eye over the 
code seen thus far in this chapter. 
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This is Ie ?y • 



try: 

with open('myfile.txt' ) as fh: 

file_data = fh.read() 
print(file_data) 
except FileNotFoundError: 

print('The data file is missing.') 
except PermissionError: 

print('This is not allowed.') 
except Exception as err: 

print('Some other error occurred:', str (err)) 


import mysql.connector 

class ConnectionError(Exception): 
pass 

class CredentialsError(Exception): 
pass 

class SQLError(Exception): 
pass 

class UseDatabase: 

def _init_(self, config: dict): 

self.configuration = config 

def _enter_(self) -> 'cursor': 

try: 

self.conn = mysql.connector.connect(**self.configuration) 
self.cursor = self.conn.cursor() 
return self.cursor 

except mysql.connector.errors.InterfaceError as err: 
raise ConnectionError(err) 

except mysql.connector.errors.ProgrammingError as err: 
raise CredentialsError(err) 

def _exit_(self, exc_type, exc_value, exc_traceback): 

self.conn.commit() 
self.cursor.close() 
self.conn.close() 

if exc type is mysql.connector.errors.ProgrammingError: 

raise SQLError(exc_value) 
elif exc_type: 

raise exc type(exc value) 


This is the 

ex£ epiion- 

sdvvy 
v &rsioy\ o-p 
£^B£i*.py w 
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Th,s is the vereioh of "vseav-dhfweb.py 

J, that "•*« y° u ' r useirs wai-fc... 


from flask import Flask, render_template, request, escape, session 
from flask import copy current request context 

from vsearch import search41etters 


from DBcm import UseDatabase, ConnectionError, CredentialsError, SQLError 
from checker import check_logged_in 

from time import sleep 


app = Flask( name ) 


app.config['dbconfig'] = {'host': '127.0.0.1', 

'user': 'vsearch', 

'password': 'vsearchpasswd', 

'database': 'vsearchlogDB', } 


@app.route('/login') 
def do_login() -> str: 

session['logged_in'] = True 
return 'You are now logged in.' 

@app.route('/logout' ) 
def do_logout() -> str: 

session.pop('logged_in') 
return 'You are now logged out.' 


1-t’s pvobably a good idea t© 
ycoitti ibis V.th” state«>ey>t 
m the sa»»e way as^ 

«you protedted the Vrth # 
statement Vie>/__t^ e —°5 
( OY\ the «e*t page)- 


@app.route('/search4' , methods=['POST']) 
def do_search() -> 'html': 

@copy current_request_context 

def log_request(req: 'flask_request', res: str) -> None: 

sleep(15) # This makes log_request really slow... 

with UseDatabase(app.config['dbconfig']) as cursor: 

_SQL = """insert into log 

(phrase, letters, ip, browser_string, results) 
values 



/9^0 9-o 9-o 9-o 9- 0 \ n ii n 
\ O kD f o o ^ o o ^ o o ^ oo/ 

cursor.execute(_SQL, (req.form['phrase'], 

req.form['letters'], 
req.remote_addr, 
req.user agent.browser, 
res, )) 


phrase = request.form['phrase'] 
letters = request.form['letters'] 
title = 'Here are your results:' 


The rest of "do_seavdh" is 

at the top of the hext page- -^ 
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results = str(search41etters (phrase, letters)) 
try: 

log_request(request, results)) 
except Exception as err: 

print('***** Logging failed with this error:', str (err)) 
return render_template('results.html', 

the_title=title, 
the_phrase=phrase, 
the_letters=letters, 
the_results=results, ) 

@app.route ( ' / ' ) 

@app.route('/entry') 
def entry_page() -> 'html': 

return render_template('entry.html', 

the title='Welcome to 



This is *thc 
v-est *thc 

-fur\<kbiorv 

search41etters on the web!') 


@app.route('/viewlog' ) 

@check_logged_in 
def view_the_log() -> 'html': 
try: 

with UseDatabase(app.config['dbconfig']) as cursor: 

_SQL = """select phrase, letters, ip, browser_string, results 
from log""" 
cursor.execute(_SQL) 
contents = cursor.fetchall () 

# raise Exception("Some unknown exception.") 

tities = ('Phrase', 'Letters', 'Remote_addr', 'User_agent', 'Results') 

return render_template('viewlog.html', 

the_title='View Log', 
the_row_titles=titles, 
the_data=contents, ) 
except ConnectionError as err: 

print ('Is your database switched on? Error:', str(err)) 
except CredentialsError as err: 

print('User-id/Password issues. Error:', str(err)) 
except SQLError as err: 

print('Is your query correct? Error:', str(err)) 
except Exception as err: 

print('Something went wrong: ', str (err)) 
return 'Error' 


app.secret_key = 'YouWillNeverGuessMySecretKey' 

if _name__ == '_main_' : 

app.run(debug=True) 
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Your code cari sometimes take a long time to execute. 

Depending on who notices, this may or may not be an issue. If some code takes 30 
seconds to do its thing “behind the scenes,” the wait may not be an issue. However, if 
your user is waiting for your application to respond, and it takes 30 seconds, everyone 
notices. What you should do to fix this problem depends on what you’re trying to do (and 
who’s doing the waiting). In this short chapter, we’ll briefly discuss some options, then look 
at one solution to the issue at hand: what happens if something takes too long? 


this is a new chapter 
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write wait read wait 


Waiting: What to Veii 


When you write code that has the potential to make your users wait, you need to 
think carefully about what it is you are trying to do. Let 5 s consider some points of 
view. 






O 

0 



Maybe it is the case that waiting for a write is different from waiting for a read, 
especially as it relates to how your webapp works? 

Let 5 s take another look at the SQL queries in log_request and view_the 
log to see how you 5 re using them. 


462 Chapter 11 3 A 




a little bit of threading 


How Are You Queryiwg Your Patabase? 

In the log_request function, we are using an SQL INSERT to add details of 
the request to our backend database. When log_request is called, it waits 
while the INSERT is executed by cursor . execute: 


def log_request(req: 'flask_request', res: str) -> None: 

with UseDatabase(app.config['dbconfig']) as cursor: 

_SQL = """insert into log 

(phrase, letters, ip, browser_string, results) 
values 

/ 9 -o 9 -o 9 -o 9 -o 9 -o \ ” ” ” 

^ O O ^ O O /- o o ^ o o ^ o o ^ 



A -1 -tWis tV»e 

Vt yiaits * or ^ 
kukend database to 


cursor.execute( SQL, 


(req.form['phrase'], 
req.form['letters'], 
req.remote_addr, 
req.user agent.browser, 
res, )) 



Geefc B 


dio its tVimOj- 


The same holds for the view_the_log function, which also waits whenever 
the SQL SELECT query is executed: 


@app.route('/viewlog') 
@check_logged_in 
def view_the_log() -> 'html 1 : 
try: 


Code that waits for something 
external to complete is referred 
to as "blocking code," in that the 
execution of your program is 
blocked from continuing until 
the wait is over. As a general 
rule, blocking code that takes a 
noticeable length of time is bad. 


Y^uv- webapp 
blotks” here, 
too, while it 
waits -for the 

databa 


ise. 


with UseDatabase(app.config['dbconfig']) as cursor: 

_SQL = """select phrase, letters, ip, browser_string, results 
from log""" 
cursor.execute( SQL) 


contents = cursor.fetchall() 
tities = ('Phrase', 'Letters', 'Remote_addr', 'User_agent', 'Results') 

return render_template('viewlog.html', 

the_title='View Log', 
the_row_titles=titles, 
the_data=contents,) 

except ConnectionError as err: .. r i» 

To «, «'« ~t *«*>» 



t TW *•* 

stili 5 oCS 


Both functions block. However, look closely at what happens after the call 
to cursor . execute in both functions. In log_request, the cursor . 
execute call is the last thing that function does, whereas in view_the_log, 
the results of cursor . execute are used by the rest of the function. 

Let’s consider the implications of this difference. 
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insert is not select 


Patabase INSERTs and SELECTs Are Pifferewt 


If you’re reading the title to this page and thinking “Of course they are!”, be assured 
that (this late in this book) we haven’t lost our marbles. 


Yes: an SQL INSERT is different from an SQL SELECT, but, as it relates to your use of 
both queries in your webapp, it turns out that the INSERT in log_request doesn’t 
need to block, whereas the SELECT in view_the_log does, which makes the 
queries very different. 

This is a key observation. 

If the SELECT in view_the_log doesn’t wait for the data to return from the 
backend database, the code that follows cursor . execute will likely fail (as itTl have 
no data to work with). The view_the_log function must block, as it has to wait for 
data before proceeding. 

When your webapp calls log_request, it wants the function to log the details of 
the current web request to the database. The calling code doesn’t really care when this 
happens, just that it does. The log_request function returns no value, nor data; the 
calling code isn’t waiting for a response. All the calling code cares about is that the web 
request is logged eventually. 


Which begs the question: why does log_request force its callers to wait? 



O 


Are you about to suggest that 
the "log_request" code could 
somehow run concurrently 
with the webapp's code? 


Yes. That’s our madcap idea. 

When users of your webapp enter a new 
search, they coukhft care less that the request 
details are logged to some backend database, 
so let 5 s not make them wait while your webapp 
does that work. 

Instead, let’s arrange for some other process to 
do the logging eventually and independently of 
the webapp’s main function (which is to allow 
your users to perform searches). 
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Poiwg More Than One Thing at Once 

Here i the plan: you’re going to arrange for the log_request function to 
exeeute independently of your main webapp. To do this, you’re going to adjust 
your webappi code so each call to log_request runs concurrently. This will 
mean that your webapp no longer has to wait for log_request to complete 
before servicing another request from another user (i.e., no more delays). 

If log_request takes an instant, a few seconds, a minute, or even hours to 
exeeute, your webapp doesn’t care (and neither does your user). What you care 
about is that the code eventually exeeutes. 


Concurrent code: you have options 


When it comes to arranging for some of your applicationi code to run 
concurrently, Python has a few options. As well as lots of support from third-party 
modules, the Standard library comes with some built-in goodies that can help here. 

One of the most well known is the threading library, which provides a high- 
level interface to the threading implementation provided by the operating system 
hosting your webapp. To use the library, all you need to do is import the 
Thread class from the threading module near the top of your program code: 


from threading import Thread 


Go ahead and add this line of code near the top of your vsearch4web . py file. 


For the lull list of (and 
all the details ahout) 
Pytho n’s Standard 
library concurrency 
options, see https:// 
docs.python.org/ 3/ 
lihrary/ concurrency. 

html. 


Now the fun starts. 


To create a new thread, you create a Thread object, assigning the name of the 
function you want the thread to exeeute to a named argument called target, 
and providing any arguments as a tuple to another named argument called args. 
The created Thread object is then assigned to a variable of your choosing. 

As an example, leti assume that you have a function called execute_slowly, 
which takes three arguments, which weil assume are three numbers. The code 
that invokes execute_slowly has assigned the three values to variables called 
glacial, plodding, and leaden. Herei how execute_slowly is invoked 
normally (i.e., without our worrying about concurrent exeeution): 


exeeute slowly(glacial, plodding, leaden) 


If execute_slowly takes 30 seconds to do what it has to do, the calling code 
blocks and waits for 30 seconds before doing anything else. Bummer. 
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(ret Puwwed Out: Use Threads 


In the big scheme of things, waiting 30 seconds for the execute_slowly 
function to complete doesn’t sound like the end of the world. But, if your user is 
sitting and waiting, theyil be wondering what’s gone wrong. 


If your application can continue to run while execute_slowly goes about 
its business, you can create a Thread to run execute_slowly concurrently. 
Here’s the normal function call once more, together with the code that turns the 
function call into a request for threaded exeeution: 




V. 


-the» direate a »ew 'Thread" obiedt, 
whidh ide»iities -the target k Jbol to 
exedite as well as a*y arguat values. 


Granted, this use of Thread looks a little strange, but it’s not really. The key 
to understanding what’s going on here is to note that the Thread object has 
been assigned to a variable (t in this example), and that the execute_slowly 
function has yet to exeeute. 

Assigning the Thread object to t prepares it for exeeution. To ask Python’s 
threading technology to run execute_slowly, start the thread like this: 

VVhe» you dall "start", the WW 
assodiied vjith the "t” thread is sdheduled 
tor e*edutio» by the "thread.^ module- 



At this point, the code that called t. start continues to run. The 30-second 
wait that results from running execute_slowly has no effect on the calling 
code, as execute_slowly’s exeeution is handled by Python’s threading 
module, not by you. The threading module conspires with the Python 
interpreter to run execute_slowly eventually. 
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When it comes to calling log_request in your webapp code, there's 
only one place you need to look: in the do_search function. Recall that 
youVe already put your call to log_request inside a try/except to 
guard against unexpected runtime errors. 

Note, too, that we've added a 15-second delay—using sleep (15) —to 
our log_request code (making it slow). Here's the current code to 

do search: 


ttevVs how Y ou 

C.ufv-erctly mvoke 


@app.route('/search4', methods=['POST']) 
def do_search() -> 'html’: 

phrase = request.form['phrase'] 

letters = request.form['letters'] 

title = 'Here are your results:' 

results = str(search41etters(phrase, letters)) 

try: 

log_request(request, results) 
except Exception as err: 

print ('***** Logging failed with this error 
str(err)) 

return render_template('results.html', 

the_title=title, 
the_phrase=phrase, 
the_letters=letters, 
the results=results,) 


r 


Add the tKv-eadm^ tode 
you'd use -bo even-tuall^f 
eitetuie "lo^jre^est • 



We are going to assume that you have already added f rom threading 
import Thread to the top of your webapp 7 s code. 

Grab your pencil, and in the space provided below, write the code you'd 
insert into do_search instead of the Standard call to log_request. 

Remember: you are to use a Thread object to run log_reques t, just 
like we did with the execute slowly example from the last page. 
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threading at work 



When it comes to calling log_request in your webapp code, there's 
only one place you need to look: in the do_search function. Recall that 
youVe already put your call to log_request inside a try/except to 
guard against unexpected run-time errors. 

Note, too, that we've added a 15 second delay - using sleep ( 15 ) - to 
our log_request code (making it slow). Here's the current code to 

do search: 


HevVs how Y ou 
tuvvently 'mvoke 
"loov-e^uest • 


@app.route('/search4', methods=[’POST']) 
def do_search() -> 'html': 

phrase = request.form['phrase'] 

letters = request.form['letters'] 

title = 'Here are your results:' 

results = str (search41etters(phrase, letters)) 

try: 

log_request(request, results) 
except Exception as err: 

print ('***** Logging failed with this error 
str(err)) 

return render_template('results.html', 

the_title=title, 
the_phrase=phrase, 
the_letters=letters, 
the results=results,) 


r 


WeVe kee\>M the "tvy” 
statement (tov now)- 


tvy- 


We assumed that you had already added from threading import 
Thread to the top of your webapp's code. 

In the space provided below, you were to write the code you'd insert into 
do_search instead of the Standard call to log_request. 

You were to use a Thread object to run log_request, just like we did 
with the recent execute slowly example. 


The "e*tept" suite 
is untha^ed, s o we 
avent showmg it heve- 


t — Thvead(tav<jet—log^ve^uest, avy—(ve<\uest» vesults)) 




Oust like the eavliev e*a»?le, identify the 
tavvt Wtior, to vun, su ?? ly any av^ents 
it needs, and don't Uyi to sthedule youv 

•tKvcadi *to v-uy\. 
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With these edits applied to vsearch4web. py, you are ready for another test run. What you’re 
expecting to see here is next-to-no wait when you enter a search into your webapp’s search page (as 
the log_request code is being run concurrently by the threading module). 

Go ahead and give it a go. 

Sure enough, the instant you click on the “Do it!” button, your webapp returns with your results. The 
assumption is that the threading module is now executing log_request, and waiting however 
long it takes to run that function’s code to completion (approximately 15 seconds). 


Takc 

a look 
“fehis 



Lo*b (j|) 
move 

messays 

hev-e 



You’re just about to give yourself a pat on the back (for a job well done) when, out of nowhere and 
after about 15 seconds, your webapp’s terminal window erupts with error messages, not unlike these: 

The las-b ve^uest wa s a suteess. 

127.0.0.1 - - [29/Jul/2016 19:43:31] "POST /search4 HTTP/1.1" 200 - 
Exception in thread Thread-6: 

Traceback (most recent call last): 

File "vsearch4web.not.slow.with.threads.but.broken. py", line 42, in log request 
cursor.exeeute(_SQL, (req.form['phrase'], 

File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/ 
werkzeug/local.py", line 343, in getattr 


raise RuntimeError( request ctx err msg) 
RuntimeError: Working outside of request context. 


I/Vhoops/ uh£au<jh-fc exdep-fcioh. 


This typically means that you attempted to use functionality that needed 
an active HTTP request. Consuit the documentation on testing for 
information about how to avoid this problem. 

During handling of the above exception, another exception occurred: 

Traceback (most recent call last): 

File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/threading.py", 
line 914, in bootstrap inner 
self.run() 


RuntimeError: Working outside of request context. 


ano-thev onc---yikcs > 


I 


This typically means that you attempted to use functionality that needed 
an active HTTP request. Consuit the documentation on testing for 
information about how to avoid this problem. 


If you check your backend database, you ll leam that the details of your web request were not logged. 
Based on the messages above, it appears the threading module isn’t at all happy with your code. 
A lot of the second group of traceback messages refer to threading. py, whereas the first group 
of traceback messages refer to code in the werkzeug and f lask folders. What’s ciear is that 
adding in the threading code has resulted in a huge mess. What's going on? 
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what’s going on? 


First Things First: Pont Panio 

Your first instinct may be to back out the code you added to run log_request in its 
own thread (and get yourself back to a known good state). But let’s not panic, and let’s 
not do that. Instead, let 5 s take a look at that descriptive paragraph that appeared twice 
in the traceback messages: 


This typically means that you attempted to use functionality that needed 
an active HTTP request. Consuit the documentation on testing for 
information about how to avoid this problem. 


This message is coming from Flask, not from the threading module. We know this 
because the threading module couldni care less about what you use it for, and 
definitely has no interest in what you’re trying to do with HTTP. 

Let 5 s take another look at the code that schedules the thread for execution, which we 
know takes 15 seconds to run, as that’s how long log_request takes. While you’re 
looking at this code, think about what happens during that 15 seconds: 


@app.route ( '/search4', methods=['POST']) 
def do_search() -> 'html': 

phrase = request.form['phrase'] 

letters = request.form['letters'] 

title = 'Here are your results:' 

results = str(search41etters(phrase, letters)) 

try: 


r 

VVV^a-t ha??e» s 

nmWiIc tWis 
tWread takes 
I 1 ? setonds “to 


> 


t = Thread(target=log_request, args=(request, 
t.start() 
except Exception as err: 

print ('***** Logging failed with this error:' 
return render_template('results.html', 

the_title=title, 
the_phrase=phrase, 
the_letters=letters, 
the results=results, ) 


results)) 


str (err)) 


The instant the thread is scheduled for execution, the calling code (the do_search 
function) continues to exeeute. The render_template function exeeutes (in the 
blink of an eye), and then the do_search function ends. 

When do_search ends, all of the data associated with the function (its context ) is 
reclaimed by the interpreter. The variables request, phrase, letters, title, 
and results cease to be. However, the request and results variables are 
passed as arguments to log_request, which tries to access them 15 seconds later. 
Unfortunately, at that point in time, the variables no longer exist, as do_search has 
ended. Bummer. 
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Pont (ret Puwwed Out: Flask Can Help 


Based on what youVe just learned, it appears the log_request function (when 
executed within a thread) can no longer “see” its argument data. This is due to 
the fact that the interpreter has long since cleaned up after itself, and reclaimed 
the memory used by these variables (as do_search has ended). Specifically, the 
re que st object is no longer active, and when log_request goes looking for it, 
it can’t be found. 


So, what can be done? Don’t fret: help is at hand. 



There’s really no need for a rewrite. 

At first glance, it might appear that you’d need to rewrite 
log_request to somehow rely less on its arguments... 
assuming that’s even possible. But it turns out that Flask 
comes with a decorator that can help here. 

The decorator, copy_current_request_context, 
ensures that the HTTP request that is active when a 
function is called remains active even when the function is 
subsequently executed in a thread. To use it, you need to 
add copy_current_request_context to the list of 
imports at the top of your webapp’s code. 

As with any other decorator, you apply it to an existing 
function using the usual @ syntax. However, there is a 
caveat: the function being decorated has to be defined 
within the function that calls it; the decorated function must 
be nested inside its caller fas an inner function). 



Here’s what we want you to do (after updating the list of imports from Flask): 

1. Take the log_request function and nest it inside the do_search function. 

2. Decorate log_request with @copy current_request_context. 

3. Confirm that the runtime errors from the last Test Drive have gone away. 
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panic no more 



(&e*ctse 

§OU>tlOM 


We asked you to do three things: 

1. Take the log_request function and nest it inside the do_search function, 

2. Decorate log_request with @copy_current_request_context. 

3. Confirm that the runtime errors from the last Test Drive have gone away. 

Here’s what our do_search code looks like after we perform tasks 1 and 2 (note: we’ll 
discuss task 3 over the page): 


@app.route('/search4', methods=['POST']) 
def do search() -> 'html': 



Task Z. The de^ov-atav* has been 
applicd bo "log^v-e^ues-t” 


Task I • The ^ 

* s * OY/ 
dc^mcd UtsitA) 
mside the u do_ 
scav"6h 


AII o-f -the 

^cs-t ot this 

tode V"Cry^aihS 

uhehahged. 


@copy_current request_context 
def log_request(req: 'flask_request', res: str) -> None: 

sleep(15) # This makes log_request really slow... 
with UseDatabase(app.config['dbconfig']) as cursor: 

_SQL = MMM insert into log 

(phrase, letters, ip, browser_string, results) 
values 

/ o. o 9- o 9-0 9-0 o. o \ H 11 1! 

\ O kD f OOf o o ^ o o ^ oo/ 

cursor.exeeute(_SQL, (req.form['phrase'], 

req.form['letters'], 
req.remote_addr, 
req.user agent.browser, 
res, )) 

phrase = request.form['phrase'] 

letters = request.form['letters'] 

title = 'Here are your results:' 

results = str(search41etters(phrase, letters)) 

try: 

t = Thread(target=log_request, args=(request, results)) 
t.start () 

except Exception as err: 

print('***** Logging failed with this error:', str(err)) 
return render_template('results.html', 

the_title=title, 
the_phrase=phrase, 
the_letters=letters, 
the results=results,) 


tliere jare no 

Dumb Questipns 


O: 


Does it stili make sense to protect the threaded invocation of log request with try/except? 


Not if you are hoping to react to a runtime issue with log_request, as the try/except will have ended before the thread 
starts. However, your system may fail trying to create a new thread, so we figure it cani hurt to leave try/except in do search. 
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Tqst Drii/q- 

Task 3: Taking this latest version of vsearch4web. py for a spin confirms that the runtime errors 
from the last Test Drive are a thing of the past. Your webapp’s terminal window confirms that all is well: 


127.0.0.1 - - 
127.0.0.1 - - 
127.0.0.1 - - 
127.0.0.1 - - 
127.0.0.1 - - 


[30/Jul/2016 20:42:46] 
[30/Jul/2016 20:43:10] 
[30/Jul/2016 20:43:14] 
[30/Jul/2016 20:43:17] 
[30/Jul/2016 20:43:37] 


'GET / HTTP/1.1" 200 - 
'POST /search4 HTTP/1.1" 200 
'GET /login HTTP/1.1" 200 - 
'GET /viewlog HTTP/1.1" 200 ■ 
'GET /viewlog HTTP/1.1" 200 ■ 


4 


Mo mere sdavy rwtime exdepW AH those Z00, 
«ea« all .s well wiih your webapp. And, 15 sedo*ds 
a+tev- you subdit a «ew seardh, youv- webapp 

ihe deia,ls ** 'I™ ba£ke " £l database 
Wl ' WUI your webapp user io wait © 



According to this card, I get to 
ask one last question. Is there any 
downside to def ining "log_request" 
within "do search"? 


No. Not in this case. 

For this webapp, the log_request function 
was only ever called by do_search, so nesting 
log_request’s within do_search isn’t an 
issue. 

If you later decide to invoke log_request 
from some other function, you may have an 
issue (and youdl have to rethink things). But, for 
now, you’re golden. 


you are here ► 
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one last review 


Is Your Webapp Robust Now? 

Here are the four questions posed at the start of Ghapter 11: 


o 

What happens 

o 

Is our webapp 

o 

What happens 

o 

What happens 


if the database connection fails? 
protected from web attacks? 
if something takes a long time? 
if a function call fails? 


Your webapp now handles a number of runtime exceptions, thanks to your use 
of try/except and some custom exceptions that you can raise and catch as 
required. 

When you know something can go wrong at runtime, fortify your code against 
any exceptions that might occur. This improves the overall robustness of your 
application, which is a good thing. 

Note that there are other areas where robustness could be improved. You spent 
a lot of time adding try/except code to view_the_log’s code, which took 
advantage of the UseDatabase context manager. UseDatabase is also used 
within log_request, and should probably be protected, too (and doing so is 
left as a homework exercise for you). 

Your webapp is more responsive due to your use of threading to handle a task 
that has to be performed eventually, but not right away. This is a good design 
strategy, although you do need to be careful not to go overboard with threads: the 
threading example in this chapter is very straightforward. However, it is very easy 
to create threading code that nobody can understand, and which will drive you 
mad when you have to debug it. Use threads with care. 

In answering question 3 —what happens if something takes a long time ?—the use of 
threads improved the performance of the database write, but not the database 
read. It is a case of just having to wait for the data to arrive after the read, no 
matter now long it takes, as the webapp wasn’t able to proceed without the data. 

To make the database read go faster (assuming it 5 s actually slow in the first place), 
you may have to look at using an alternative (faster) database setup. But that’s a 
worry for another day that we won’t concern ourselves with further in this book. 

However, having said that, in the next and last chapter, we do indeed consider 
performance, but we’11 be doing so while discussing a topic everyone understands, 
and which weVe already discussed in this book: looping. 



474 Chapter 11 3 A 





a little bit of threading 


Chapter 11 %'$ Code, 1 of 2 


This is the latest a«d gv-eatest 
^ veirsion of "vseavch^web.py". 


from flask import Flask, render_template, request, escape, session 
from flask import copy current request context 
from vsearch import search41etters 

from DBcm import UseDatabase, ConnectionError, CredentialsError, SQLError 
from checker import check_logged_in 

from threading import Thread 


from time import sleep 



app = Flask( name ) 

app.config['dbconfig'] = 

{'host': '127 

.0.0.1', 


'user': 'vsearch'. 


'password': 

'vsearchpasswd' 

@app.route('/login' ) 
def do login() -> str: 

'database': 

'vsearchlogDB', 

session['logged in'] 

= True 



return 'You are now logged in.' 

@app.route('/logout' ) 
def do_logout() -> str: 

session.pop('logged_in') 
return 'You are now logged out.' 

@app.route('/search4', methods=['POST']) 
def do_search() -> 'html': 

@copy current_request_context 

def log_request(req: 'flask_request', res: str) -> None: 

sleep(15) # This makes log_request really slow... 
with UseDatabase(app.config['dbconfig']) as cursor: 

_SQL = MMM insert into log 

(phrase, letters, ip, browser_string, results) 
values 

/ o, O, O, O, O. 0 \ WWW 

^ O O f O O f O O ^ OOf o o / 

cursor.execute(_SQL, (req.form['phrase'], 

req.form['letters'], 
req.remote_addr, 
req.user agent.browser, 
res, )) 

phrase = request.form['phrase'] 
letters = request.form['letters'] 
title = 'Here are your results:' 


The <resi of "do_sea<rth" is 

at the top of ihe nexi pag e .-^ 


you are here ► 
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the code 


Chapter 11 3 A'$ Code, 2 of 2 


results = str(search41etters(phrase, letters)) 
try: 


t = Thread(target=log_request, args=(request, 
t.start () 

except Exception as err: 

print('***** Logging failed with this error:' 
return render_template('results.html', 

the_title=title, 
the_phrase=phrase, 
the_letters=letters, 
the results=results,) 


results)) 


str(err)) 



TWis is -bKc 


@app.route ( ' / ' ) 

@app.route('/entry') 

def entry_page() -> 'html': 

return render_template('entry.html', 

the title='Welcome to 


v-est o£ 
u do_scavd^ w 

search41etters on the web! 


) 


@app.route('/viewlog' ) 

@check_logged_in 
def view_the_log() -> 'html': 
try: 

with UseDatabase(app.config['dbconfig']) as cursor: 

_SQL = """select phrase, letters, ip, browser_string, results 
from log""" 
cursor.execute(_SQL) 
contents = cursor.fetchall () 

# raise Exception("Some unknown exception.") 

tities = ('Phrase', 'Letters', 'Remote_addr', 'User_agent', 'Results') 

return render_template('viewlog.html', 

the_title='View Log', 
the_row_titles=titles, 
the_data=contents,) 
except ConnectionError as err: 

print('Is your database switched on? Error:', str(err)) 
except CredentialsError as err: 

print('User-id/Password issues. Error:', str(err)) 
except SQLError as err: 

print('Is your query correct? Error:', str(err)) 
except Exception as err: 

print('Something went wrong:', str(err)) 
return 'Error' 

app.secret_key = 'YouWillNeverGuessMySecretKey' 

if _name__ == '_main_' : 

app.run(debug=True) 
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12 adVanced fteratlon 


# Looping Like Crazy + 



lt’s often amazing how much time our programs spend in loops. 

This isrYt a surprise, as most programs exist to perform something quickly a whole heap 
of times. When it comes to optimizing loops, there are two approaches: (1) improve the 
loop syntax (to make it easier to specify a loop), and (2) improve how loops execute (to 
make them go faster). Early in the lifetime of Python 2 (that is, a long, long time ago), the 
language designers added a single language feature that implements both approaches, 
and it goes by a rather strange name: comprehension. But don’t let the strange name 
put you off: by the time you’ve worked through this chapter, you’ll be wondering how you 
managed to live without comprehensions for so long. 


this is a new chapter 
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flights data 


Pahawas Puzzers Have Places to Go 

To learn what loop comprehensions can do for you, you’re going to take a look at 
some “real” data. 

Operating out of Nassau on New Providence Island, Bahamas Buzzers provides 
island-hopping flights to some of the larger island airports. The airline has 
pioneered just-in-time flight scheduling: based on the previous day 5 s demand, the 
airline predicts (which is just a fancy term for “guesses”) how many flights they 
need the next day. At the end of each day, the BB Head Office generates the next 
day 5 s flight schedule, which ends up in a text-based GSV {comma-separated value) file. 



Here’s what tomorrow 5 s GSV file contains: 



This is a Standard C£\/ &le> 
with the *P»rst Ime a»ven over 
t© header in-Pormation- It ali 
looks extept the -faet 
that everythinj s UPPERCA^t 
(whidh is a litti e w old sC\\oo\ l )• 


TIME,DESTINATION 
09:35,FREEPORT 
17:00,FREEPORT 
09:55,WEST END 
19:00,WEST END 
10:45,TREASURE CAY 
12:00,TREASURE CAY 
11:45,ROCK SOUND 
17:55,ROCK SOUND 



The header telis us to 
expe^t two dolumns o-P data-* 
one representing times, the 
other destinations. 


The rest o-P the 
C£\/ -Pile Contains 


data- 



Head Office calls this GSV file buzzers . csv. 

If you were asked to read the data from the GSV file and display it on screen, 
you’d use a with statement. Here’s what we did at IDLE’s >>> prompt, after 
using Python 5 s os module to change into the folder that contains the file: 


Python 3.5.2 Shell 


£et this 
to the 
-Polder 

youVe 

usin<y 


»> 

>> import os 

»> os.chdir( '/Users/paul/buzzdata 1 ) 

»> 

»> with open (' buzzers.csv' ) as raw_data: 
print (raw_data.read()) 


TIME, 

09:35 

17:00 

09:55 

19:00 

10:45 

12:00 

11:45 

17:55 

»> I 


DESTINATION 
,FREEPORT 
,FREEPORT 
,WEST END 
,WEST END 
,TREASURE CAY 
,TREASURE CAY 
,ROCK SOUND 
,ROCK SOUND 


4- 


The raw 
CSI/ data 

-Prom the 
-Pile- 


Ln: 59 Coi: 4 


dii o+ the 

«hav-ac-tek-s in the 

tile ih one go. 




Learn more about the CSV format here: 
h ttps://en. wikipedia. org/wiki/Commo- 
separated_values. 
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advanced iteration 


Reading CSV Pata As Usts 

The CSV data, in its raw form, is not very useful. It would be more useful if you 
could read and break apart each line at the comma, making it easier to get at the 
data. 

Although it is possible to do this “breaking apart” with hand-crafted Python code 
(taking advantage of the string objecfs split method), working with CSV data 
is such a common activity that the Standard library comes with a module named 
csv that can help. 

Here 5 s another small for loop that demonstrates the csv module in action. 
Unlike the last example, where you used the read method to grab the entire 
contents of the file in onego, in the code that follows, csv . reader is used to read 
the CSV file one line at a time within the for loop. On each iteration, the for loop 
assigns each line of CSV data to a variable (called line), which is then displayed 
on screen: 


This looks betbev: 
eadh Ime of da-ta 
-(Vom "the CSV 

-file has been 
•tuvmed m*to a list 




»> with open ( 'buzzers . csv' ) as data: 

for line in csv.reader(data): 



print (line) 


['TIME', 

'DESTINATION'] 


['09:35', 

'FREEPORT'] 


['17:00', 

'FREEPORT'] 


['09:55', 

'WEST END'] 


['19:00', 

'WEST END'] 


['10:45' , 

'TREASURE CAY'] 


['12:00', 

'TREASURE CAY'] 


['11:45' , 

'ROCK SOUND'] 


['17:55', 

'ROCK SOUND'] 


»> 



>>> 



Ln: 77 Coi: 4 


...*then v-ead 
•the data a Ime 
at a time with 

w tsv.v-eadev- w . 


The csv module is doing quite a bit of work here. Each line of raw data is being 
read from the file, then “magically” turned into a two-item list. 

In addition to the header information (from the flrst line of the file) being 
returned as a list, each individual flight time and destination pair also gets its own 
list. Take note of the type of the individual data items returned: everything is a 
string, even though the first item in each list (clearly) represents a time. 

The csv module has a few more tricks up its sleeve. Another interesting function 
is csv. DictReader. Let 5 s see what that does for you. 


you are here ► 
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csv to dictionary 


Readiwg CSV Pata As Pictiowaries 


Here’s code similar to the last example, but for the fact that this new code uses csv . 
DictReader as opposed to csv. reader. When DictReader is used, the data 
from the CSV file is returned as a collectiori of dictionaries, with the keys for each 
dictionary taken from the CSV file 5 s header line, and the values taken from each of the 


subsequent lines. Here’s the code: 



The 


Python 3.5.2 Shell 


»> 

»> 


»> with open ( 'buzzers . csv' ) as data: 

for line in csv. Di|ctReader (data) 
print (line) 



{'DESTINATION' 
{'DESTINATION’ 
'DESTINATION' 
'DESTINATION' 
'DESTINATION' 
'DESTINATION' 
{'DESTINATION' 
{'DESTINATION' 
»> 

»> 


FREEPORT 


'TIME': '09:35' } 

'FREEPORT', 'TIME': '17:00'} 

'WEST END', 'TIME': '09:55'} 

'WEST END', 'TIME': '19:00'} 

' TREASURE CAY', 'TIME': '10:45'} 
’TREASURE CAY', 'TIME': '12:00'} 
'ROCK SOUND', 'TIME': '11:45'} 

'ROCK SOUND', 'TIME': '17:55'} 


A\ 


The values 



: 79 Coi: 19 


Usihg "csv.DieiReadev" is a 

s '7 r le but it makes a biq 

di++eirehde- IVhat was L e s of 

bsMlast time) ave ^ | ihes of 

diexiohdv-ies. 


TIME,DESTINATION 
09:35,FREEPORT 
17:00,FREEPORT 
09:55,WEST END 
19:00,WEST END 
10:45,TREASURE CAY 
12:00,TREASURE CAY 
11:45,ROCK SOUND 
17:55,ROCK SOUND 



like tbis. 


There is no doubt that this is powerful stuff: with a single call to DictReader, the 
csv module has transformed the raw data in your CSV file into a collection of Python 
dictionaries. 

But imagine youVe been tasked with converting the raw data in the CSV file based on 
the following requirements: 


o 

o 


Convert the flight times from 24-hour format to AM/PM format 


Convert the destinations from UPPERCASE to Titlecase 


In and of themselves, these are not difficult tasks. However, when you consider the raw 
data as a collection of lists or a collection of dictionaries, they can be. So, let’s write 
a custom for loop to read the data into a single dictionary that can then be used to 
perform these conversions with a lot less fuss. 


480 Chapter 12 

















advanced iteration 


Let'$ Pack Up a Little Pit 

Rather than use csv. reader or csv. DictReader, let 5 s roll our own code 
to convert the raw data in the CSV file into a single dictionary, which we can then 
manipulate to perform the required conversions. 

We Ve had a chat with the Head Office folks over at Bahamas Buzzers , and they Ve told us 
theyVe very happy with the conversions we have in mind, but would stili like the data 
kept in its “raw form,” as that’s how their antiquated departures board expects its data 
to arrive: 24-hour format for flight times, and all UPPERGASE for destinations. 

You could perform conversions on the raw data in your single dictionary, but let’s 
ensure that the conversions are performed on copies of the data, not the actual raw 
data as read in. Although it’s not totally ciear at the moment, the noises coming out of 
Head Office seem to indicate that whatever code you create may have to interface with 
some existing Systems. So, rather than face the prospect of converting the data back 
into its raw form, let’s read it into a single dictionary as is, then convert to copies as 
required (while leaving the raw data in the original dictionary untouched). 

It’s not an awful lot of work (over and above what you had to do with the csv module) 
to read the raw data into a dictionary. In the code below, the file is opened, and the 
first line is read and ignored (as we don’t need the header info). A for loop then reads 
each line of raw data, splitting it in two at the comma, with the flight time being used 
as your dictionary key , and the destination used as your dictionary value. 


The v3v/ d3t3 



TIME,DESTINATION 
09:35,FREEPORT 
17:00,FREEPORT 
09:55,WEST END 
19:00,WEST END 
10:45,TREASURE CAY 
12:00,TREASURE CAY 
11:45,ROCK SOUND 
17:55,ROCK SOUND 

A\ 


Cdv\ you bv-eak e3£h 
line ih buo, usihj) 
"the domma 3s the 
delimitev? 


“the 
-Ciie 3$ 
befove- 

Cveste 3 
em?ty didtio*3vy 
dalled “flight* • 


pis^lay “the dontent* 
of the didWsvy, 
v/hidh looks 3 little 

wessed wtil- 



•the “pvetty- 
pvmtmg” Iibv3vy 
pvodudes move 
human—fviendly 
output- 


r 




\ 






Python 3.5.2 Shell 


»> with open ( 'buzzers . csv' ) as data: 
ignore = data.readline() 
flights = {} 
for line in data: 

k, v = line.split( 

Pvodess_ y flights [k] = v 

C3th Ime- KT 

»> flights 

{'12:00': 'TREASURE CAY\n', '09:35 

17:00': 'FREEPORT\n' , '19:00': 'WEST END\n 

: 'ROCK SOUND\n', '10:45': 'TREASURE CAY\n 
: 'WEST END\n' , '11:45': 'ROCK SOUND\n'} 

»> 

»> import pprint 
»> pprint .pprint (flights) 


/g^ove the 
he3dev ih-Po. 


Assia h destih3tioh 
to -flight time- 

FREEPORT\n', ' 

'17:55' 


09:55 


{'09:35' 

'FREEPORT\n', 

'09:55' 

'WEST END\n', 

'10:45' 

'TREASURE CAY\n', 

'11:45' 

'ROCK SOUND\n', 

'12:00' 

'TREASURE CAY\n', 

'17:00' 

'FREEPORT\n', 

'17:55' 

'ROCK SOUND\n', 

'19:00' 

'WEST END\n'} 


The mdlusio* 
of the y\ev/Vme 

dh3v3dtev looks 

a little str3*ge> 
doesnt >t? 


»> 


Ln: 486 Coi: 4 


Btreak apavt ih e 
liy>e at the do*»**a, 
whidh tretuirns Iwo 
v ali»es- -the key 
tflight time) and 
v alue ^destinationi 


you are here ► 
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no to newlines 


Strippmg, Thew Splittiwg, Your Raw Pata 

The latest with statement used the split method (included with all string 
objects) to break the line of raw data in two. What’s returned is a list of strings, 
which the code individually assigns to the k and v variables. This multivariable 
assignment is possible due to the fact that you have a tuple of variables on the left 
of the assignment operator, as well as code that produces a list of values on the 
right of the operator (remember: tuples are immutable lists): 


A tuple o-P k, 

vaHabl es 

on the ^ 

left 


v = line.split( 



Code that produtes a list 
o-f values on the ri^ht 


Another string method, strip, removes whitespace from the beginning and end 
of an existing string. Let’s use it to remove the unwanted trailing newline from the 
raw data before preforming the split. 

Here 5 s one final version of our data-reading code. We create a dictionary called 
f lights, which uses the flight times as keys and the destinations (without the 
newline) as values: 



• • • Python 3.5.2 Shell 

»> 

»> with open ( 'buzzers . csv' ) as data: 
ignore = data.readline() 
flights = {} 
for line in data: 


Whitespace: the 

following characters are 
considered whitespace 
in strings: space, \t, \n, 
and \r. 


k, v = line.strip().split( ', ' ) 
flights[k] = v 


»> 



»> pprint. pprint (flights) 


{'09:35' 
'09:55' 
'10:45' 
'11:45' 
' 12 : 00 ' 
'17:00' 
'17:55' 
'19:00' 
»> 

»> 


'FREEPORT', 
'WEST END', 
'TREASURE CAY 
'ROCK SOUND', 
'TREASURE CAY 
'FREEPORT', 

'ROCK SOUND', 
'WEST END'} 


This Co&t s-fo-ips ihc 
line, ihen spliis i-fc, io 
pvodude ihe daia in 
ihe -foV"n»ai \rc«\uivcd. 


What if you switched the order of the methods in your code, like so: 


line.split(', f ).strip() 


What do you think would happen? 



When you string methods 
t^etheir like this, it's called 
a method chaih." 


You may not have 
s P°tted this, but the 
order of the vows in 
the dictionary different 
from what s in the data ■ 
file- This happens because 
dictionaries do NOT 
maintain insertion order. 

Pon't worry about this 
ioir now. 


TIME,DESTINATION 
09:35,FREEPORT 
17:00,FREEPORT 
09:55,WEST END 
19:00,WEST END 
10 : 45,TREASURE CAY 
12:00,TREASURE CAY 
11:45,ROCK SOUND 
17:55,ROCK SOUND 
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advanced iteration 


Pe Careful Wben Chaining Method Calis 

Some programmers don’t like the fact that Python’s method calls can be chained 
together (as strip and split were in the last example), because such chains can be 
hard to read the first time you see them. However, method chaining is popular among 
Python programmers, so you’11 likely run across code that uses this technique “in the 
wild.” Care is needed, however, as the order of the method calls is not interchangeable. 

As an example of what can go wrong, consider this code (which is very similar to what 
came before). Whereas before the order was strip, then split, this code calls 
split first, then tries to call strip. Look what happens: 


TVie 

is «at V>av?7’ a ' r><1 _ 

tvasbes vjftVi a* J ^ 


• • • Python 3.5.2 Shell 

»> 

»> 

»> with open ( 'buzzers.csv' ) as data: 
ignore = data.readline() 
flights = {} 
for line in data: 

k, v = line.splitstrip() 
flights[k] = v 


Traceback (most recent call last): 

File "<pyshell#194> n , line 5, in <module> 
k, v = line.splitstrip() 

AttributeError: 'list' object has no attribute 'strip' 
»> 

»> 


Ln: 607 Coi: 4 




T\)c order 
this me-thod 
^haih has 
-Plipped 

whait you uscd 
bc-Po\rc. 


To understand what’s going on here, consider the type of the data to the right of the 
assignment operator as the above method chain executes. 

Before anything happens, line is a string. Calling split on a string returas a list 
of strings, using the argument to split as a delimiter. What started out as a string 
(line) has dynamically morphed into a list , which then has another method invoked 
against it. In this example, the next method is strip, which expects to be invoked on 
a string, not a list, so the interpreter raises an AttributeError, as lists don’t have a 
method called strip. 

The method chain from the previous page does not suffer from this issue: 

• • • 

line.strip () .split ') 



With this code, the interpreter starts out with a string (in line), which has any 
leading/trailing whitespace removed by strip (yielding another string), which 
is then split into a list of strings based on the comma delimiter. There’s no 
AttributeError, as the method chain doesn’t violate any typing rules. 


you are here ► 
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transforming data 


Transforming Pata into the Format You Need 


Now that the data is in the f lights dictionary, let 5 s consider the data manipulations 
BB Head Office has asked you to perform. 

The first is to perform the two conversions identified earlier in this chapter, creating a 
new dictionary in the process: 


o 

o 


Convert the flight times from 24-hour format to AM/PM format 
Convert the destinations from UPPERCASE to Titlecase 


Applying these two transformations to the f lights dictionary allows you to turn the 
dictionary on the left into the one on the right: 


The -flijhf tirr.es 

^ *" e tweried 4or« 
2-^-howr -fov-mat, " 




{'09:35' : 

'FREEPORT’, 

{'09:35AM': 

'Freeport', 

'09:55' : 

'WEST END', 

'09:55AM': 

'West End' , 

'10:45' : 

'TREASURE CAY', 

'10:4 5AM' : 

'Treasure Cay 

'11:45' : 

'ROCK SOUND', 

'11:45AM': 

'Rock Sound', 

'12:00' : 

'TREASURE CAY', 

'12:00PM': 

'Treasure Cay 

'17:00' : 

'FREEPORT', 

'05:00PM': 

'Freeport', 

'17:55' : 

'ROCK SOUND', 

'05:55PM': 

'Rock Sound', 

'19:00' : 

'WEST END'} 

'07:00PM': 

'West End' } 


The destinations 
^ ave fconvev-ted 

u uppeRCA^e 

to Title^sc* 


Note that the data in both dictionaries has the same meaning, it 5 s just the 
representation that’s changed. Head Office needs the second dictionary, as they feel that 
its data is more universally understandable, as well as friendlier; Head Office thinks all- 
UPPERGASE is akin to shouting. 

At the moment, the data in both dictionaries has a single line for each flight time/ 
destination combination. Although Head Office will be happy when you Ve transformed 
the dictionary on the left into the dictionary on the right, they Ve also suggested that 
it would be really useful if the data could be presented with single destinations as keys 
and a list of flight times as values—that is, a single row of data for each destination. 
Let’s look at how that dictionary would appear before embarking on coding the 
required manipulations. 
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advanced iteration 


Transforming into a Pictionary Of lists 

Once the data in f lights has been transformed, Head Office wants you to perform 
this second manipulation (discussed at the bottom of the last page): 


{'0 9:35AM' : 
'09:55AM' 
'10:45AM' 
'11:45AM' 

'12:00PM' 

'05:00PM' 
'05:55 PM' 

'07:00PM' 


wcv-c valucs 
b tCotr\t 


West End', 
Treasure Cay', 
Rock Sound', 
Treasure Cay' 
Freeport', 

Rock Sound', 
West End' } 



{'Freeport': ['09:35AM', '05:00PM'], 

'Rock Sound': ['11:45AM', '05:55PM'], 

'Treasure Cay': ['10:45AM', '12:00PM'] 

'West End': ['09:55AM', '07:00PM']} 


VVV^art >«eve kevs 

bctomc lists ot values. 





r 


Thiwk about the data wrangling that s weeded here... 

There 5 s a bit of work required to get from the raw data in the GSV file to the 
dictionary of lists shown above on the right. Take a moment to think about how you’d 
go about doing this using the Python you already know. 

If you’re like most programmers, it won’t take you long to work out that the for loop 
is your friend here. As Python’s main looping mechanism, for has already helped you 
extract the raw data from the GSV file and populate the f lights dictionary: 


with open('buzzers.csv') as data: 
ignore = data.readline() 
flights = {} 
for line in data: 

k, v = line.strip().split(',') 
flights[k] = v 

Ifs tempting to suggest that this code be amended to perform the transformations to 
the raw data as it is read from the GSV file—that is, prior to adding rows of data to 
flights. But recall the Head Office’s request that the raw data remain untouched in 
flights: any transformations need to be applied to a copy of the data. This makes 
things more complex, but not by much. 


TWis is a tlassit «se ot 

"V\ and a Wl Hfk 

'd' 0 " m P f ^ 


you are here t 
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basic converts first 


lets Po the Pasic Conversions 

At the moment, the f lights dictionary contains flight times in 24-hour format as its 
keys, with UPPERGASE strings representing destinations as its values. You have two 
initial conversions to perform: 


o 

o 


Convert the flight times from 24-hour format to AM/PM format 
Convert the destinations from UPPERCASE to Titlecase 


Gonversion #2 is easy, so let 5 s do that one first. Once data is in a string, simply call the 
string 5 s title method, as this IDLE session demonstrates: 


The w *t»'tlc w 

rr\ e-thod vc-tuv^s 

a topy o-f -bKe 
data m s . 



»> s = "I DID NOT MEAN TO SHOUT. " 


»> print(s) 

I DID NOT MEAN TO SHOUT. 
»> t = s . title () 

»> print(t) 

I Did Not Mean To Shout. 


f ,s is 7 ^ 4iendl 
thah be+ore. 


iClr 


Gonversion #1 involves a bit more work. 


If you think about it for a minute, things get quite involved when it comes to 
converting 19:00 into 7:00 PM. However, this is only the case when you look at the 
19:00 data as a string You’d need to write a lot of code to do the conversion. 

If you instead consider that 19 : 0 0 is a time, you can take advantage of the 
dat e time module that is included as part of Python’s Standard library. This module 5 s 
datet ime class can take a string (like 19:00) and convert it to its equivalent AM/ 
PM format using two prebuilt functions and what’s known as string format specifiers. 
Here’s a small function, called convert2ampm, which uses the facilities of the 
dat e time module to perform the conversion you need: 



For more on string 
lormat specifiers, 
see littos://docs. 
nytkon.org/ 3/ 
likrary/ ia te time, 
ktmltstrf time-ani- 

strptime-kekavior. 


from datetime import datetime 

def convert2ampm(time24: str) -> str: 

return datetime.strptime(time24, 1 %H:%M 1 ).strftime( 1 %I:%M%p 1 ) 
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advanced iteration 



Lefs put the conversion techniques from the last page to work. 

Below is the code that reads the raw data from the CSV file, populating 
the f lights dictionary as it goes.The convert2ampmfunction is also 
shown. 

Your job is to write a for loop that takes the data in f lights and 
converts the keys to AM/PM format, and the values to Titlecase. A new 
dictionary, called f lights2, is created to hold the converted data. Use 
your pencil to add the for loop code in the space provided. 

Hint: when processing a dictionary with a for loop, recall that the 
items method returns the key and value for each row (as a tuple) on each 
iteration. 


Pe-Cme the 

dorwevs \ oy \ 'fufiC-tiofi- 



from datetime import datetime 
import pprint 


def convert2ampm(time24: str) -> str 
return datetime.strptime(time24, 


'%H:%M').strftime( ' %I:%M%p') 



the data 
-(Vom the -Pile- 


with open(’buzzers.csv’) as data: 
ignore = data.readline() 
flights = {} 
for line in data: 

k, v = line.strip () .split ( ’, ') 
flights[k] = v 



pprint.pprint(flights) 
print () 


flights2 = {} ^r 


P perWg the 7 


The dicbonary, ta\\t& 

u %htsZ", starts 


/\dd youv- w £o/-> 

loop heve- 


pprint.pprint(flights2) 



Pvetiy-pvmi tv>e ditWavy b> 

do^irrm that the donvevsions ave wovki^. 


you are here ► 
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convert then run 


Sharpen your pencil 

Solution 



We sdved all 
dodc 

m a -file talled 

U do___6o^r\VCV't^y , 


The w ’i‘tc^s 
mrthod v-c-tuvns 

C 36 ^ VOV/ -&TOW» 

-the U %M* 

d'\thor\^- 


Your job was to write a for loop that takes the data in f lights and 
converts the keys to AM/PM format, and the values to Titlecose. You were 
to create a new dictionary, called f lights2, to hold the converted data, 
and you were to add the for loop code in the space provided. 



from datetime import datetime 
import pprint 


def convert2ampm(time24: str) -> str: 

return datetime.strptime(time24, '%H:%M').strftime('%I:%M%p') 


with open ( 'buzzers.csv') as data: 
ignore = data.readline() 
flights = {} 
for line in data: 

k, v = line.strip () .split ( ', ') 
flights[k] = v 

pprint.pprint(flights) 
print () 



flights2 = {} 

-Co\r k, v m ^Ii i£c^sO : 

rf"!*'???P. 1 ??.?. t? “ v "WJeO 

pprint.pprint(flights2) 



0 * itc\ratioh, -the 

key C\v\ k ) is £ohve\rfced 

io AM/PM &v*ai 

^Ch UScd as -the hCW 
did-fciohdlrys key. 


The value (i* v ^ ,s 
dovwev"tedi io Trtledase, 
assijynedi io the 




If you exeeute the above program, two dictionaries are displayed on screen (which we’re showing 
below, side by side). The conversions work, although the ordering in each dictionary differs, as the 
interpreter does not maintain insertion order when you populate a new dictionary with data: 


TWis »s 


{'09:35' 

’FREEPORT', 

{'05:00PM’: 

’Freeport’, 

'09:55' 

'WEST END', 

'05:55PM' 

'Rock Sound', 

'10:45' 

'TREASURE CAY', 

'07:00PM' 

'West End', 

'11:45' 

'ROCK SOUND', 

'09:35AM' 

'Freeport', 

'12:00' 

'TREASURE CAY', 

' 09:55AM' 

'West End', 

'17:00' 

'FREEPORT', 

'10:45AM’ 

'Treasure Cay', 

'17:55' 

'ROCK SOUND', 

’11:45AM’ 

'Rock Sound', 

'19:00' 

'WEST END'} 

’12:00PM' 

'Treasure Cay'} 
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advanced iteration 


Pid You Spot the Pattern in Your Code? 

Take another look at the program you Ve just executed. There ’s a very common 
programming pattern used twice in this code. Gan you spot it? 

from datetime import datetime 

import pprint 

def convert2ampm(time24: str) -> str: 

return datetime.strptime(time24, ’%H:%M 1 ).strftime( 1 %I:%M%p 1 ) 

with open('buzzers.csv') as data: 
ignore = data.readline() 
flights = {} 
for line in data: 

k, v = line.strip().split(',') 
flights[k] = v 

pprint.pprint(flights) 

print() 

flights2 = {} 

for k, v in flights.items(): 

flights2[convert2ampm(k)] = v.title() 

pprint.pprint(flights2) 

If you answered: “the for loop,” you’re only half-right. The for loop is part of the 
pattern, but take another look at the code that surrounds it. Spot anything else? 


from datetime import datetime 
import pprint 


def convert2ampm(time24: str) -> str: 

return datetime.strptime(time24, '%H:%M')•strftime('%I:%M%p 1 ) 


£ath the 
loofs is pv-eteded by 
the ereatio* ot a new, 
cmfty data sbruh^t 
U a dietior\av-y). 


with open('buzzers.csv') as data: 
ignore = data.readline() 
flights = { } 
for line in data: 

k, v = line.strip().split ) 
flights[ k] = v 

pprint.pprint(flights) 
print () 



Eadh o( the »c « i - , 

. u , m loo P s ^tes 

f,, toAt *hat adds data 

to the hew data stn,dture, 

based o, the p«rodessi, 9 ^ S or» e 

existmg data. 


flights2 = {} 

for k, v in flights.items(): 

flights2[convert2ampm(k)] = v.title() 



pprint.pprint(flights2) 


you are here ► 
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spot the pattern 


Spotting the Pattern with lists 

The examples on the last page highlighted the programming pattern as it relates to 
dictionaries: start with a new, empty dictionary, then use a for loop to process an 
existing dictionary, generating data for a new dictionary as you go: 


The new, 

mitially empty, 

diitionav-y 





flights2 = {} 
for k, v in flights.items(): 

flights2[convert2ampm(k)] = v.title() 



prodesses -the 
existing data. 


The existing data is used 
to generate keys and 
values, whieh are inserted 
'mto the new dictionary. 


The existing dictionary 


This pattern also makes an appearance with lists, where it is easier to spot. Take a 
look at this IDLE session, where the keys (i.e., the flight times) and the values (i.e., the 
destinations) are extracted from the f lights dictionary as lists, then converted into 
new lists using the programming pattern (numbered 1 through 4 in the annotations): 


|. £tart 
With a new, 
empty list 


A - - \/iew the 
new listis — 

data. 

I- Start 

w 'th a new, 
e**pty Ii st. 


Af. \/iew the 
new lists 

data. 





Python 3.5.2 Shell 


»> 

»> 

»> flight_times = [] 

»> for ft in f lights . keys () : 




flight times.append(convert2ampm(ft)) 


»> print (f light_times) 

[ ' 05:00PM', '09:55AM', '11:45AM', '10:45AM', '07:00PM', '05:55PM', 

'12:00PM', 1 09:35AM'] 

»> _ 

»> destinations = [] 

»> for dest in f lights . values () : 

destinations.append(dest.title()) 



Ln: 154 Coi: 4 


2- Iterate 
through each o-P 
the -flight times. 


»> print (destinations) 

['Freeport', 'West End', 'Rock Sound', 'Treasure Cay', 'West End', 

'Rock Sound', 'Treasure Cay', 'Freeport'] 

»> 

»> I 



3. Append the 
donverted data 
to the new list- 

2-. Iterate 
through eadh of 
the destinations. 


Append the 

toYv/erlcd data 

to the new list. 


This pattern is used so often that Python provides a convenient shorthand notation for 
it called the comprehension. Let’s see what 5 s involved in creating a comprehension. 
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advanced iteration 


Cohvertiwg Patterws iwto Comprehensions 


Let 5 s take the most recent for loop that processed the destinations as our 
example. Here it is again: 


destinations = [] 
for dest in flights.values(): 

destinations.append(dest.title()) 

K._ 



I- Sia?- 1 
*'th a hew, 
c^p-fey Ii st 


2 ~ Iterate throu^h 
eath of -bV^c 
destinations. 


j’ the donverted 

d^ta to the new list. 


Python’s built-in comprehension feature lets you rework the above three lines 
of code as a single line. 


To convert the above three lines into a comprehension, we’re going to step 
through the process, building up to the complete comprehension. 


Begin by starting with a new, empty list, which is assigned to a new variable (which 
we’re calling more_dests in this example): 


more dests 



|. S-tavt wth a *ew, 
en\ list (and Jive 
it a na«>e) 


Specify how the existing data (in f lights in this example) is to be iterated over 
using the familiar for notation, and place this code within the new list 5 s square 
brackets (note the absence of the colon at the end of the for code): 


more dests = [for dest in flights.values ()] 


2-- Iterate thvoujjh eadh 
°f the destinations. 




f\(ote that 
theve s NO 
dolon here- 


To complete the comprehension, specify the transformation to be applied to the 
data (in dest), and put this transformation before the for keyword (note the 
absence of the call to append, which is assumed by the comprehension): 


more dests 


[dest.title() for dest in flights.values()] 

the donvevted data to the new 
list> without aetually £allin^ append • 


And that’s it. The single line of code at the bottom of this page is functionally 
equivalent to the three lines of code at the top. Go ahead and run this line of code 
at your >>> prompt to convince yourself that the more_dests list contains the 
same data as the destinations list. 


you are here ► 
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alternative to for 


Take a Closer Look at the Comprehension 

Let 5 s look at the comprehension in a little more detail. Here’s the original three 
lines of code as well as the single-line comprehension that performs the same task. 

Remember: both versions produce new lists (destinations and more_ 
dests) that have exactly the same data: 

destinations = [] 

for dest in flights.values(): 

destinations.append(dest.title()) 

\ 

more_dests = [dest.title() for dest in flights.values()] 

It’s also possible to pick out the parts of the original three lines of code and see 
where theyVe been used in the comprehension: 


more de 



If you spot this pattern in other code, you can easily turn it into a comprehension. 
For example, here 5 s some code from earlier (which produces the list of AM/PM 
flight times) reworked as a comprehension: 


f ts2 


flight_times = [] 
for ft in flights.keys () : 

flight_times.append(convert2ampm(ft)) 

i 

= [convert2ampm(ft) for ft in flights.keys()] 



These do “the 
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advanced iteration 


What'$ thc Pig Peal? 



These comprehensions look hard to 
understand. I'm pretty happy using a "for" 
loop when I need to do some+hing like this. Is 
learning how to write comprehensions really 
worth the eff ort? 


Yes. We think it’s well worth the effort. 

There are two main reasons why taking the time to 
understand comprehensions pays off. 

Firstly, as well as requiring less code (which means 
comprehensions are easier on your fingers), the Python 
interpreter is optimized to run comprehensions as 
quickly as possible. This means comprehensions 
execute faster than the equivalent for loop code. 

Secondly, comprehensions can be used in places where 
for loops can’t. In fact, youVe already seen this, as all 
the comprehensions presented so far in this chapter 
have appeared to the right of the assignment operator, 
which is something a regular for loop can’t do. This 
can be surprisingly useful (as you’11 see as this chapter 
progresses). 


Comprehensions arent jost for lists 

The comprehensions youVe seen so far have created new lists, so each is known 
as a list comprehension (or listcomp for short). If your comprehension creates 
a new dictionary, it’s known as a dictionary comprehension ( dictcomp ). 

And, so as not to leave any data structure out, you can also specify a set 

comprehension (. setcomp ). 

There 5 s no such thing as a tuple comprehension ; we’11 explain why later in this chapter. 
First, though, let’s take a look at a dictionary comprehension. 


you are here ► 
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dictionary comprehensions 


Specifyiwg a Pictiowary Comprehension 

Recall the code from earlier in this chapter that read the raw data from the GSV 
file into a dictionary called f lights. This data was then transformed into a new 
dictionary called f lights 2, which is keyed by AM/PM flight times and uses 
“titlecased” destinations as values: 


flights2 = {} 

for k, v in flights.items(): 
flights2[convert2ampm(k)] 


v.title () 



This dode £oirfo\rr»is "fco 
^ompirchchsioh patier*. 


Let 5 s rework these three lines of code as a dictionary comprehension. 

Start by assigning a new, empty dictionary to a variable (which we are calling 
more_f lights): 

more_flights = {} 

Specify how the existing data (in f lights) is to be iterated over using the for 
loop notation (being sure not to include the usual trailing colon): 

more_flights = {for k, v in flights.items()} 

Ko“tc “that 
-theve s NO 

dolon heve- 

To complete the dictcomp, specify how the new dictionaryh keys and values 
relate to each other. The for loop at the top of the page produces the key by 
converting it to an AM/PM flight time using the convert2ampm function, 
while the associated value is turned into titlecase thanks to the stringi title 
method. An equivalent dictcomp can do the same thing and, as with listcomps, 
this relationship is specified to the left of the dictcomp’s for keyword. Note the 
inclusion of the colon separating the new key from the new value: 

more_flights = {convert2ampm(k): v.title() for k, v in flights.items()} 

3. /\ssodia*te the donvev-ted key with 
fb w titledased w value (and no-le the 
use o( "the dolon heve). 

And there it is: your first dictionary comprehension. Go ahead and take it for a 
spin to confirm that it works. 



2-. Iterate through eadh 
the keys and values 
•from the existing data- 


I. stavt >with a nex, 
empty dittionavy- 
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Extend Comprehensioni with Filters 

Let 5 s imagine you need only the converted flight data for Freeport. 

Reverting to the original for loop, you’d likely extend the code to include an if 
statement that filters based on the current value in v (the destination), producing code 
like this: 


just_freeport = {} 
for k, v in flights.items(): 
if v == 'FREEPORT': 

just freeport[convert2ampm(k)] 


= v.title() 


The tlight data is only tonvevted 
and added to the >t>ee?ort 
dittionary it it velates to the 
Freeport destination- 


If you execute the above loop code at the >>> prompt, you’11 end up with just two 
rows of data (representing the two scheduled flights to Freeport as contained in the 
raw data file). This shoukkft be surprising, as using an i f in this way to filter data is 
a Standard technique. It turns out that such filters can be used with comprehensions, 
too. Simply take the i f statement (minus the colon) and tack it onto the end of your 
comprehension. Here 5 s the dictcomp from the bottom of the last page: 


TIME,DESTINATION 
09:35,FREEPORT 
17:00,FREEPORT 
09:55,WEST END 
19:00,WEST END 
10:45,TREASURE CAY 
12:00,TREASURE CAY 
11:45,ROCK SOUND 
17:55,ROCK SOUND 



The v-aw 


data 


more flights = {convert2ampm(k): v.title() for k, v in flights.items()} 


And here’s a version of the same dictcomp with the filter added: 


just freeport2 = {convert2ampm(k): v.title() for k, v in flights.items() if v == 'FREEPORT'} 


If you execute this filtered dictcomp at your >>> prompt, the data in the newly 
created j ust_f reeport2 dictionary is identical to the data in j ust_f reeport. 
Both j ust_f reeport and j ust_f reeport2’s data is a copy of the original data 
in the flights dictionary. 

Granted, the line of code that produces j ust_f reeport2 looks intimidating. 

Many programmers new to Python complain that comprehensions are hard to 
read. However, recall that Python’s usual end-of-line-means-end-of-statement rule is 
switched off whenever code appears between a bracket pair, so you can rewrite any 
comprehension over multiple lines to make it easier to read, like so: 

just_freeport3 = {convert2ampm(k) : v.title () 

for k, v in flights.items() 
if v == 'FREEPORT'} 



t 

The tlight data is only 
Converted and added 
io ihe just_-freeport2 w 
dictionary i-f ii relates io 
the Freeport destination. 


/<xa'II need to get used 
io reading those one-line 
Comprehensions. That said, 
Pythoh pV*og\rdr*r*eV“S frt 
indreasingly writing longer 
Comprehensions over multiple lines 
30 '1°* 11 s « ^is syntax, too). 


you are here * 
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a quick review 


Recall What You Set Out to Po 


Now that you Ve seen what comprehensions can do for you, let’s revisit the required 
dictionary manipulations from earlier in this chapter to see how we’re doing. Here 5 s 
the first requirement: 

_^ Convehi 4or« 

hou\T (orn,ai io 

A/W/P/V) (oTmdi. 



{'09:35' : 

'FREEPORT', 

{'09:35AM': 

'Freeport', 

'09:55' : 

'WEST END', 

'09:55AM': 

'West End', 

'10:45' : 

'TREASURE CAY', 

'10:4 5AM' : 

'Treasure Cay', 

'11:45' : 

'ROCK SOUND', 

'11:45AM': 

'Rock Sound', 

'12:00' : 

'TREASURE CAY', 

'12:00PM': 

'Treasure Cay' 

'17:00' : 

'FREEPORT', 

'05:00PM': 

'Freeport', 

'17:55' : 

'ROCK SOUND', 

'05:55PM': 

'Rock Sound', 

'19:00' : 

'WEST END'} 

'07:00PM': 

'West End' } 



Convev-t trom 




—=? WPPfcRCASfc to-" 




Trtletase- 



Given the data in the f lights dictionary, you Ve seen that the following dictionary 
comprehension performs the above conversions in one line of code , assigning the copied 
data to a new dictionary called fts here: 

fts = {convert2ampm(k): v.title() for k, v in flights.items()} 


The second manipulation (listing flight times per destination) is a little more involved. 
There 5 s a bit more work to do due to the fact that the data manipulations are more 
complex: 

Wnat wcmc values 


{'0 9:35AM' 
'09:55AM' 
'10:45AM' 
'11:45AM' 
'12:00PM' 
'05:00PM' 
'05:55 PM' 
'07:00PM' 



Freeport', 

'West End' , 

1 Treasure Cay 
1 Rock Sound', 
'Treasure Cay' 
1 Freeport', 

1 Rock Sound', 
'West End' } 



'Rock Sound': ['11:45AM', '05:55PM'], 

'Treasure Cay': ['10:45AM', '12:00PM'], 

'West End': ['09:55AM', '07:00PM']} 



VVV,a-t 'weve kevs 
betome lists ot values- 
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advanced iteration 


^Jharpen your pencil 


Before starting to work on the second manipulation, lefs pause for a bit to see 
how well the comprehension material is seeping into your brain. 


YouVe been tasked with transforming the three for loops on this page into 
comprehensions. As you do, don't forget to test your code in IDLE (before 
flipping the page and peeking at our Solutions). In fact, before you try to write 
the comprehensions, execute these loops and see what they do. Write your 
comprehension Solutions in the spaces provided. 



data 


[ 1, 2, 3, 


4, 5, 6, 7, 8 ] 


evens = [] 

for num in data: 

if not num % 2: 

evens.append(num) 



The h operator is Pytho„'s rnodu | 0 
operator, whidh Works as -follows: aiv eh 
two numbers, divide the £rst by the 
%ttonA > the» retur» the re»>ai»der. 


© 



data = [ 1, 'onei 2 , 1 two ’ r 3, 

words = [] 

for num in data: 

if isinstance(num, str): 
words.append(num) 


three 


4, 1 four 1 ] 



TV^ 


ci-wV’ Y>\? thedks to 


em» \aI 





data = list('So long and thanks for ali the fish' .split ()) 

title = [] 

for word in data: 

title.append(word.title() ) 


you are here ► 
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comprehending the Solutions 



You were to grab your pencil, and pop your thinking cap on. For each 
of these three for loops, you were tasked with transforming them 
into comprehensions, being sure to test your code in IDLE. 


o 


data = [ 1, 2, 3, 4, 5, 3,1,3} 

evens = [] 

for num in data: 

if not num % 2: 

evens.append(num) 



T"hesc tou\r lihcs ot loop £ode 
(whidh populate "evens") bedo**e 
one line o+ dompvehension. 


CVChS — C KUm -fov- mn\ in data i-f Y\o t 


hurm % 2- 3 



o 


data = [ 1, 'onei 2, ' two' , 3, ’ three' , 4, ' four' ] 


words = [] 

for num in data: 

if isinstance(num, str): 
words.append(num) 



/\ojain, tWts W— Ime loop 
is v-ewov-ked as a one— 

1'mC tompvehension. 


y/o\rds ==■ [ num -fo\r 


num m 


data i-P isinstandefnum, stv") 3 




data = list('So long and thanks for ali the fish’.split()) 


title = [] 


for word in data: 

title.append(word.title()) 




You should tihd this 
oy\c the easiest ot the 
three fas it tontains ho 
tilter). 


title — L wordtitleO -Cor word 'm data 3 
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advanced iteration 


Pe al with Complexity the Python Way 

With your comprehension practice session behind you, let’s experiment at the >>> 
prompt to work out what has to happen to the data in the f t s dictionary in order to 
transform it into what 5 s required. 

Before writing any code, take another look at the required transformation. Notice how 
the keys in the new dictionary (on the right) are a list of unique destinations taken 
from the values in the f ts dictionary (on the left): 



It turns out that producing those four unique destinations is very straightforward. 
Given that you have the data on the left in a dictionary called f ts, you can access all 
of the values using f ts . values, then feed that to the set BIF to remove duplicates. 
Let’s sto re the unique destinations in a variable called dests: 


6|vaV> all ot -ty 

values m * X5 ’ 

the» W the- 

^ the "set” BIF- 
This ^ets 7°« the 
data 7 <x» 



»> dests = set (fts .values ()) 
»> print (dests) 

{'Freeport', 'West End', 'Rock 


Now that you have a way to get the unique destinations, it 5 s time to grab the flight 
times associated with those destinations. This data is also in the fts dictionary 


Sound 1 f 1 Treasure Cay 1 } 

WcJre aire (oiAir Uhi^ue 
dcs-tih^tiohS; whi^h you 
US C as -the hcw 
di^-tioha\ry's kcys. 


Before turning the page, have a think about how you’d go about extracting the flight 
times given each unique destination. 


In fact, don’t worry about extracting all the flight times for every destination; just work 
out how to do it for West End first. 
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Extract a Single Pestination's Flight Times 


Let’s start by extracting the flight time data for a single destination, namely West End. 
Here’s the data you need to extract: 



You «eed to tum 

these keys in a Ii$t 
o-f values. 



{’09:35AM': 

'Freeport', 

(** 09 : 55AMj]}) 

'West End', 

'10:45AM': 

'Treasure Cay', 

'11:4 5AM' : 

'Rock Sound', 

'12:00PM': 

'Treasure Cay' 

'05:00PM': 

'Freeport', 

'05:55PM': 

'Rock Sound', 

' 07 : 00PM^ 

'West End' } 


As before, pull up the >>> prompt and get to work. Given the f ts dictionary, you can 
extract the West End flight times using code like this: 


I. Stav-t wi*fch a 
Kew, empty list- 

Piltcir -the data 
Oh destihatioh 
'West E*d" 


|t wovked! Heres 
tke data you heed- 


»> wests = [] 

»> for k, v in fts.items() : 

if v == 'West End': 

wests.append(k) 


»> print(wests) 

['09:55AM', '07:00PM'] 


JT- 


2- Extract the keys 
and values -fv-om the 
“tts" didtionav-y. 

A". Affe^d the "West 
End" tliOjht times to 
the "uiests” list- 


On seeing this code, you should hear little alarm bells ringing in your brain, as this 
for loop is surely a candidate for reworking as a list comprehension, right? 

That for loop becomes this equivalent listcomp: 

»> wests2 = [k for k, v in fts.items() if v == 


Wkat was -fouv lihes o-P 
tode is now one, thanks to 
youir use ot a listdomp. 



It also wov-ked' 
Heres the — 
data you heed- 


»> print (wests2) 

['09:55AM', '07:00PM'] 
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Now that you know how to extract this data for one 
specific destination, let’s do it for all the destinations. 
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Extract Fligbt Times for AII Pestinations 

You now have this code, which extracts the set of unique destinations: 

dests = set (fts. values ()) ^- The uni^ue 

destinations 

And you also have this listcomp, which extracts the list of flight times for a given 
destination (in this example, that destination is West End): 

wests2 = [k for k, v in fts.items() if v == 'West End'] £ 


The %ht Wcs 
£ov -the “West 

^nd” destination 

j 


To extract the list of flights times for ali of the destinations, you need to combine these 
two statements (within a for loop). 

In the code that follows, weVe dispensed with the need for the dests and west2 
variables, preferring to use the code directly as part of the for loop. We no longer 
hardeode West End , as the current destination is in de st (within the listcomp): 



The uhi^ue 
destihatiohs 


»> for dest in set (fts .values ()) : 

print(dest, '->', [k for k, v in fts.items() 


Treasure Cay -> ['10:45AM', '12:00PM'] 

West End -> ['07:00PM', '09:55AM'] 

Rock Sound -> ['05:55PM', '11:45AM'] 
Freeport -> ['09:35AM', '05:00PM'] 



The fl^ht 
■times for the 
destination 
referred to by 
the Current valwe 
of "dest”- 


The fact that weVe just written a for loop that appears to conform to our 
comprehension pattern starts our braufs little bell ringing again. Let’s try to suppress 
that ringing for now, as the code you Ve just experimented with at your >>> prompt 
displays the data we need...but what you really need is to store the data in a new 
dictionary. Let’s create a new dictionary (called when) to hold this newly extracted 
data. Head back to your >>> prompt and adjust the above for loop to use when: 


I. Stavt wi-bh 
a new, e^pty 
dtetionav-y. 


2-. Extract the unique 
set o-f destinations. 

»> for dest in set (fts .values ()) : 

when[dest] = [k for k, v in fts.items() if v == dest] 



»> pprint. pprint (when) 

{'Freeport': ['09:35AM', '05:00PM'], 

'Rock Sound': ['05:55PM', '11:45AM'], 
'Treasure Cay': ['10:45AM', '12:00PM'] 

'West End': ['07:00PM', '09:55AM']} 


1 . 


Weve it is-* -the 
data you heed, 
in a dietioha^y 
called “when” 


y Updaie -the 
'“hen d\cbona<ry 
wi-th the -flight 

times. 


If you’re like us, your little brain bell (that you Ve been trying to suppress) is likely 
ringing loudly and driving you crazy as you look at this code. 
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That Feeliwg You ftct... 

...when a single line of code starts to look like magic. 

Switch off your brain bell, then take another look at the code that makes up your most 
recent for loop: 


when = {} 

for dest in set(fts.values()): 

when[dest] = [k for k, v in fts.items() if v == dest] 


This code conforms to the pattern that makes it a potential target for reworking as a 
comprehension. Here 5 s the above for loop code reworked as a dictcomp that extracts 
a copy of the data you need into a new dictionary called when2: 

when2 = {dest: [k for k, v in fts.items() if v == dest] for dest in set(fts.values())} 


It looks like magic , doesn’t it? 

This is the most complex comprehension you Ve seen so far, due mainly to the fact that 
the outer dictcomp contains an inner listcomp. That said, this dictcomp showcases one 
of the features that set comprehensions apart from the equivalent for loop code: you 
can put a comprehension almost anywhere in your code. The same does not hold for 
for loops, which can only appear as statements in your code (that is, not as part of 
expressions). 

Of course, that’s not to say you should always do something like this: 


Thes t do the 

J 

when2 = {dest: [k for k, v in fts.items() if v == dest] for dest in set(fts.values())} 


when = {} 

for dest in set(fts.values()): 

when[dest] = [k for k, v in fts.items() if v == dest] 

\ 



Be warned: a dictionary comprehension containing an embedded list comprehension 
is hard to read the first time you see it. 

However, with repeated exposure, comprehensions do get easier to read and 
understand, and—as stated earlier in this chapter—Python programmers use them 
a lot. Whether you use comprehensions is up to you. If you are happier with the for 
loop code, use that. If you like the look of comprehensions, use them...just doni feel 
you have to. 
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Before moving on, let’s put all of this comprehension code into our do_convert. py file. We can 
then run the code in this file (using IDLE) to see that the conversions and transformations that are 
required by Bahamas Buzzers are occurring as required. Confirm that your code is the same as ours, 
then execute the code to confirm that everything is working to specification. 


# # # do convert.py -/Users/paul/Desktop/NewBook/ch12/do convert.py (3.5.2) 


from datetime import datetime 
import pprint 

def convert2ampm (time24: str) -> str: 

return datetime.strptime(time24, '%H:%M' ).strftime( '%I:%M%p' ) 

with open ( 'buzzers.csv' ) as data: 
ignore = data.readline() 
flights = {} 
for line in data: 

k, v = line.strip().split( ', 1 ) 
flights[k] = v 

pprint.pprint(flights) 
print () 

fts = {convert2ampm(k): v.title() for k, v in flights.items()} 

pprint.pprint(fts) 
print () 

when = {dest: [k for k, v in fts.items() if v == dest] for dest in set (fts.values())} 


pprint.pprint(when) 
print () 


Python 3.5.2 Shell 



/chl2/do 




|. The ori^inal, 
v-aw data, as 
vead 'm w 
CSV data £'«k- 
This is K {lio>Wts"- 

\ The ,is t of -flight 

tin.es per destination 

u ( f*braeied 4or» 

•fts"). This is "when 




09 

09 

10 

11 

12 

17 

17 

19 


35' 

55' 

45' 

45' 

00 ' 

00 ' 

55' 

00 ' 


convert.py ==== 
1 FREEPORT', 
'WEST END', 
'TREASURE CAY' 

'ROCK SOUND', 
'TREASURE CAY' 

'FREEPORT', 

'ROCK SOUND', 
'WEST END'} 



{'05:00 PM' 
'05:55PM' 
' 07:00PM' 
'09:35AM' 
'09:55AM' 
'10:45AM' 
'11:45AM' 
'12:00PM' 


'Freeport', 
'Rock Sound' 
'West End', 

'Freeport', 
'West End', 
'Treasure Cay', 
'Rock Sound', 
'Treasure Cay'} 


The raw 
daia, dopied and 

irahs-PoirirKed 'K-fco 

A/ty/P/VJ 

£*d Tiilcdasc. 

This is 


{'Freeport': ['05:00PM', '09:35AM'], 

'Rock Sound': ['05:55PM', '11:45AM'], 

'Treasure Cay': ['10:45AM', '12:00PM'], 

'West End': ['07:00PM', '09:55AM']} 


»> 


Ln: 214 Coi: 4 



0 
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ask questions 


tfiereiare n° 

Dumb Questions 


O: So...let me get this straight: a comprehension is just 
syntactic shorthand for a Standard looping construet? 


Yes, specifically the for loop. A Standard for loop and 
its equivalent comprehension do the same thing. It’s just that the 
comprehension tends to exeeute considerably faster. 



When will I know when to use a list comprehension? 


There are no hard and fast rules here. Typically, if you are 
producing a new list from an existing one, have a good look at your 
loop code. Ask yourself if the loop is a candidate for conversion to 
an equivalent comprehension. If the new list is “temporary” (that 
is, used once, then thrown away), ask yourself if an embedded list 
comprehension would be better for the task at hand. As a general 
rule, you should avoid introducing temporary variables into your code, 
especially if they’re only used once. Ask yourself if a comprehension 
can be used instead. 



Can I avoid comprehensions altogether? 


Yes, you can. However, they tend to see quite a bit of use 
within the wider Python community, so unless your plan is to never 
look at anyone else’s code, we’d suggest taking the time to become 
familiar with Python’s comprehension technology. Once you get used 
to seeing them, you’ll wonder how you ever lived without them. Did 
we mention that they are fast ? 


O: Yes, I get that, but is speed such a big deal nowadays? My 
laptop is super-fast and it runs my for loops quick enough. 


A- 

Jr \* That’s an interesting observation. It’s true that today we have 
computers that are vastly more powerful than anything that’s come 
before. It’s also true that we spend a lot less time trying to eke out 
every last CPU cycle from our code (because, let’s face it: we don’t 
have to anymore). However, when presented with a technology that 
offers a performance boost, why not use it? It’s a small bit of effort for 
a big return in performance. 



f 


I a really strong cup of cc 
e something in it) helps n 
around most comprehens 


coffee 


(with 


nd 


I 


little 


get 


me 


my 


a 


head 


the 


By 


lons 


they 


work 


with 


tuples? 


do 


and 


sets 


way 


That’s a great question. 

And the answer is: yes and no. 

Yes, it is possible to create and use a set 
comprehension (although, to be honest, you 
will encounter them only very rarely). 

And, no, there 5 s no such thing as a 
“tuple comprehension.” Wedl get to 
why this is after weVe shown you set 
comprehensions in action. 
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The Set Comprehension in Aetion 

A set comprehensions (or seteomp for short) allows you to create a new set in one 
line of code, using a construet that’s very similar to the list comprehension syntax. 

What sets a seteomp apart from a listcomp is that the set comprehension is 
surrounded by curly braces (unlike the square brackets around a listcomp). This 
can be confusing, as dictcomps are surrounded by curly braces, too. (One wonders 
what came over the Python core developers when they decided to do this.) 

A literal set is surrounded by curly braces, as are literal dictionaries. To teli them 
apart, look for the colon character used as a delimiter in dictionaries, as the colon 
has no meaning in sets. The same advice applies to quickly determining whether a 
curly-braced comprehension is a dictcomp or a seteomp: look for the colon. If it 5 s 
there, you’re looking at a dictcomp. If not, it 5 s a seteomp. 

Here 5 s a quick set comprehension example (which hearkens back to an earlier 
example in this book). Given a set of letters (in vowels), and a string (in 
message), the for loop as well as its equivalent seteomp produce the same 
resuit— a set of the vowels found in me s sage: 


vowels = {'a', 'e', 'i 'a ' o', ’ u'} 

message = "Don't forget to pack your towel." 


found = set() 
for v in vowels: 

if v in message: 
found.add(v) 


The sett°">? tollas 
■the sa»«e pattev* as 
the listfcomf>- 



found2 



{ v for v in vowels if v in message } 



Kote the use tuvty Wates heve, 
as this tomprehensiem pvodufces a set 
when e*efcuted ty the 'mtevpvetev 


Take a few moments to experiment with the code on this page at your >>> prompt. 
Because you already know what listcomps and dictcomps can do, getting your 
head around set comprehensions isn’t that tricky. There 5 s really nothing more to 
them than what 5 s on this page. 
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How to Spot a Comprehension 

As you become more familiar with the look of comprehension code, they become 
easier to spot and understand. Here 5 s a good general rule for spotting list 
comprehensions: 

II you spot code surrounded hy [ and J, then you are 

lookingf at a list comprehension. 

This rule can be generalized as follows: 


II you spot code surrounded hy hrackets (curly or 
scruare), then you are likely looking at a comprehension. 

Why the use of the word “likely”? 

In addition to code being surrounded by [ ], comprehensions can also, as you Ve 
seen, be surrounded by { }. When code is surrounded by [ and ], you are looking 
at a list comprehension. When code is surrounded by { and }, you are looking at 
either a set or a dictionary comprehension. A dictcomp is easy to spot thanks to 
its use of the colon character as a delimiter. 

However, code can also appear between ( and ), which is a special case , even 
though you’d be forgiven for suggesting that code surrounded by parentheses 
must surely be a tuple comprehension. You’d be forgiven, but wrong: “tuple 
comprehensions” don’t exist, even though you can put code between ( and ) . 

After the “fun” you Ve been having with comprehensions so far in this chapter, you 
may be thinking: could, this get any weirder? 



] 


I'm either a set 
or a dictionary 
comprehension. 


O 

o 


{ ...code... } 



o 

O 


( ...code... ) 


Let’s conclude this chapter (and this book) by exploring what’s going on with 
code that appears between ( and ) . It’s not a “tuple comprehension,” but it is 
obviously allowed, so what is it? 
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What About "Tuple Cowprehewsiows"? 

Pythoffs four built-in data structures (tuples, lists, sets, and dictionaries) can be put to 
many uses. However, all but tuples can be created via a comprehension. 

Why is this? 

It turns out that the idea of a “tuple comprehension” doesfft really make sense. Recall 
that tuples are immutable : once a tuple is created, it cannot be changed. This also 
means that it’s not possible to generate a tuple’s values in code, as this short IDLE 
session shows: 



Create a neu/, 
empty tuple- 


• • • Python 3.5.2 Shell 

»> 

»> names = () 

»> 

»> for n in ( ' John ' , ' Paul' , ' George ' , ' Ringo ' ) : 

Tv-y to dy^a^ieally 
add data to the 

Traceback (most recent call last): . . 

File "<pyshell#17>", line 2, in <module> tuple* 
names.append(n) 

AttributeError: 'tuple' object has no attribute 'append' 

»> 

»> 


Ln: 44 Coi: 4 




names.append(n) 


You ea^/t append 
to dh existihjj 
tuple, as it is 

immutable. 


There’s nothing weird or wonderful going on here, as this is the behavior expected 
from tuples: once one exists, it cannot be changed. This fact alone should be enough 
to rule out using a tuple within any sort of comprehension. But take a look at this 
interaction at the >>> prompt. The second loop differs from the first in the smallest 
of ways: the square brackets around the listcomp (in the first loop) have been replaced 
with parentheses (in the second): 


Python 3.5.2 Shell 



VVViat yves? Both 

loops generate 
the sa**e results- 



»> 

»> for i in [x*3 for x in [1,2,3,4,5]] 
print (i) 


3 

6 

9 

12 

15 

»> 

»> for i in (x*3 for x in [1,2,3,4,5]) 
print (i) 


3 

6 

9 

12 

15 

»> 



This -for loop and list 
eomprehe^sioh eombmatioh displays 
eatVi ot the lists values tripled- 
v/ou know this is a listdomp, as 
that’s dode inside s<\uave Wadkets. 



But look at this. The pav-entheses 
makes this look like a "tuple 
dompvehensioh"—but you know sudh 
a thing is impossible. Yet the loop 
stili pvodudes the expedted output- 
l/Veivd, eh? 


Ln: 46 Coi: 4 
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generate your data 


Parentheses Around Code ss Generator 

When you come across something that looks like a listcomp but is surrounded by 
parentheses, you’re looking at a generator: 


This looks like a 

lis-ttorop, t 

i^s a ^e^eva-tov-. 



for i in (x*3 for x in [1, 2, 3, 4, 
print (i) 



h gehera-tor tav\ 
be used ahywhcrc 
a lis-t^omp is used. 


As you saw at the bottom of the last page, when you replace a listcomp’s surrounding 
square brackets with parentheses, the results are the same; that is, the generator and 
the listcomp produce the same data. 


a*d produces -the 
same results. 


However, they do not exeeute in the same way. 


If you’re scratching your head at the previous sentence, consider this: when a listcomp 
exeeutes, it produces all of its data prior to any other processing occurring. Taken 
in the context of the example at the top of this page, the for loop doesn’t start 
processing any of the data produced by the listcomp until the listcomp is done. This 
means that a listcomp that takes a long time to produce data delays any other code 
from running until the listcomp concludes. 


With a small list of data items (as shown above), this is not a big issue. 


But imagine your listcomp is required to work with a list that produces 10 million items 
of data. You Ve now got two issues: (1) you have to wait for the listcomp to process 
those 10 million data items before doing anything else , and (2) you have to worry that the 
computer running your listcomp has enough RAM to hold all that data in memory 
while the listcomp exeeutes (10 million individual pieces of data). If your listcomp 
runs out of memory, the interpreter terminates (and your program is toast). 


(rewerators produce data items One at a time... 

When you replace your listcomp^ square brackets with parentheses, the listcomp 
becomes a generator, and your code behaves differently. 

Unlike a listcomp, which must conclude before any other code can exeeute, a 
generator releases data as soon as the data is produced by the generator 5 s code. This 
means if you generate 10 million data items, the interpreter only needs enough 
memory to hold one data item (at a time), and any code that’s waiting to consume the 
data items produced by the generator exeeutes immediately; that is, there’s no waiting. 


Listcomps and 
generat ors 
produce the 
same results, 
hut operate 
in a very 
dillerent way. 


There’s nothing quite like an example to understand the difference using a generator 
can make, so let 5 s perform a simple task twice: once with a listcomp, then again with a 
generator. 
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Using a Ustcomp to Frocess URLs 


To demonstrate the difference using a generator can make, let 5 s perform a task using a 
listcomp (before rewriting the task as a generator). 

As has been our practice throughout this book, let’s experiment with some code at 
the >>> prompt that uses the reques ts library (which lets you programmatically 
interact with the Web). Here’s a small interactive session that imports the requests 
library, defines a three-item tuple (called uris), and then combines a for loop with 
a listcomp to request each URUs landing page, before processing the web response 
returned. 

To understand what’s going on here, you need to follow along on your computer. 


Download 

"recjuests" Irom 

PyPI using the 

d. 


P 1 ? 


comman 


Petme a Uple ot WRL-s. feel W to 
substiMe V°u' r oum URU be«. Oust be 
suve to detine at least tbvee- 



The "tov" loop eontains a 
■stdorwp, whidh, -pon eadh ot 
the URLs i* Vis", gets the 
website s landing page. 


Python 3.5.2 Shell 


»> 

»> import requests 
»> 

»> uris = ( ' http://headf irstlabs . com’ , 1 http://orer/lly. com ’ , 'http://twitter.com') 
»> 

»> for resp in [requests.get(uri) for uri in uris]: 

print(len (resp.content), resp.status_code, resp.uri) 

31590 -> 200 -> http://headfirstlabs.com/ 

78722 -> 200 -> http://www.oreilly.com/ 

128244 -> 200 -> https://twitter.com/ 

>» | ^ 



Ln: 106 Coi: 4 


Wothing weiird on wondev-tul hev-e. The 
output pnoduded is exadtly whats expedted- 


VVitV» eadb v-esponse nedeived, 
display the size o( the 
netunned landing page (in 


If you’re following along on your computer, you will experience a noticeable delay 
between entering the for loop code and seeing the results. When the results appear, 
they are displayed in one go (all at once). This is because the listcomp works through 
each of the URLs in the uris tuple before making any results available to the for 
loop. The outcome? You have to wait for your output. 

Note that there’s nothing wrong with this code: it does what you want it to, and 
the output is correct. However, let’s rework this listcomp as a generator to see the 
difference it makes. As mentioned above, be sure to follow along on your computer as 
you work through the next page (so you can see what happens). 
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Usiwg a Generator to Process URLs 


Here’s the example from the last page reworked as a generator. Doing so is easy; 
simply replace the listcomp’s square brackets with parentheses: 


"Python 3.5.2 Shell 1 


>>> S- A* '"•poHawt chah 9 e: veplace 

»> I the s 9 ua\re brackets with 

»> parentheses. 

»> 4/ 

»> for resp in (requests.get(uri) for uri in uris): 

print(len (resp.content), resp.status_code, 



I _ « 


resp.uri) 


Ln: 151 Coi: 1 


A short moment after entering the above for loop, the first resuit appears: 


• • • *Python 3.5.2 Shell* 

»> 

»> 

»> for resp in (requests.get(uri) for uri in uris): 

print(len (resp.content), resp.status_code, ' , resp.uri) 

31590 -> 200 -> http://headfirstlabs.com/ ^ - The -Piv-s-fc WRiAs v-espohse 

Ln: 153 Coi: 0 


Then, a moment later, the next line of results appear: 


"Python 3.5.2 Shell 1 


»> 


»> for resp in (requests.get(uri) for uri in uris): 

print(len (resp.content), resp.status_code, resp.uri) 


31590 -> 200 -> http://headfirstlabs.com/ 
78722 -> 200 -> http://www.oreilly.com/ 


The sedond WRL*s vesponse 


Ln: 154 Coi: 0 


Then—finally 


a few moments later, the last results line appears (and the for loop ends): 


• • • Python 3.5.2 Shell 

»> for resp in (requests. get (uri) for uri in uris): 

print(len (resp.content), resp.status_code, ' , resp.uri) 

31590 -> 200 -> http://headfirstlabs.com/ 

78722 -> 200 -> http://www.oreilly.com/ p ~ > 

128244 -> 200 -> https://twitter.com/ ^-thifdh Vmdh lAKb- s fespon 

»> | 

Ln: 156 Coi: 4 
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Using a (renerator: What Just Happewed? 

If you compare the results produced by your listcomp to those produced by your 
generator, they are identicol. However, the behavior of your code isn’t. 

The listcomp waits for all of its data to be produced before feeding any data 
to the waiting for loop, whereas the generator releases data as soon as it 
becomes available. This means the for loop that uses the generator is much more 
responsive, as opposed to the listcomp (which makes you wait). 

If you’re thinking this isn’t really that big a deal, imagine if the URLs tuple was 
defined with one hundred, one thousand, or one million URLs. Further, imagine 
that the code processing the response is feeding the processed data to another 
process (perhaps a waiting database). As the number of URLs increases, the 
listcomp 5 s behavior becomes worse compared to that of the generator. 








No. We wouldn’t say that. 

Don’t misunderstand: the fact 
that generators exist is great , but 
this doesn’t mean you’11 want to 
replace all of your listcomps with 
an equivalent generator. Like a lot 
of things in programming, which 
approach you use depends on what 
you’re trying to do. 

If you can afford to wait, then 
listcomps are fine; otherwise, 
consider using a generator. 


One interesting usage of generators is to embed them within a function. Let 5 s take 
a look at encapsulating your just-created generator in a function. 
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generator functions rock 


Pefme What Your Function Necds to Po 

Let 5 s imagine that you want to take your requests generator and turn it into a 
function. You Ve decided to package the generator within a small module you’re 
writing, and you want other programmers to be able to use it without having to 
know or understand generators. 

Here’s your generator code once more: 


import requests 


|mpo\rt 5hy \rc^ui\rcd lib\rav-ics. 



uris = ('http://headfirstlabs.com', 'http://oreilly.com', 'http://twitter.com') 


for resp in (requests.get(uri) for uri in uris): 


print (len (resp.content), '->', resp.status code, '->', resp.uri) 


Petme a U?le 
of URU- 


I 


P\ro£ess thc 

gchcv-atcd data. 


The ^enevatov (v-emembev- : 
loob like a listton>p « 

swvounded by paventkeses) 



Let’s create a function that encapsulates this code. The function, which is called 
gen_f rom_urls, takes a single argument (a tuple of URLs), and returns a tuple 
of results for each URL. The returned tuple contains three values: the length of 
the URL’s content, the HTTP status code, and the URL the response came from. 

Assuming gen_f rom_urls exists, you want other programmers to be able to 
execute your function as part of a for loop, like this: 


from uri utils import gen from uris 



/mpov-t -the tuhd-tioh 
-hror* youir module. 



URU- 


uris = ('http://headfirstlabs.com', 'http://oreilly.com'. 


for 


resp_len, status, uri in gen from uris (uris) : 
print (resp len, status, uri) 



Pvodess -the data. 


Although this new code does not look all that different from the code at the top of 
the page, note that programmers using gen_f rom_urls have no clue (nor do 
they need to know) that youVe using requests to talk to the Web. Nor do they 
need to know that youVe using a generator. All of your implementation details 
and choices are hidden behind that easy-to-understand function call. 

Let 5 s see what 5 s involved in writing gen_f rom_urls so that it can generate the 
data you need. 


'http://twitter.com') 


Call the -function 

eatk itevaW 

ot tke "tov” loop 
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Yield to the Power of Generator Functions 

Now that you know what the gen_f rom_urls function needs to do, let 5 s go about 
writing it. Begin by creating a new file called url_utils . py. Edit this file, then add 
import requests as its first line of code. 

The function’s def line is straightforward, as it takes a single tuple on the way in, and 
returns a tuple on output (note how weVe included type annotations to make this 
explicit for users of our generator function). Go ahead and add the function’s def line 
to the file, like so: 

A-P-fcer 

import requests . ». 

ir*po\rtiha 

____«_/ ii 

def gen_from_urls(uris: tuple) -> tuple: 

The function’s suite is the generator from the last page, and the for line is a sii 
copy-and-paste: 

import requests 

def gen_from_urls(uris: tuple) -> tuple: 

for resp in (requests.get(uri) for uri in uris): 


mple 


hCW -fuh£-fcioh. 


Add m 7^ “W 

loop 



The next line of code needs to “return” the resuit of that GET request as performed 
by the requests . get function. Although it’s tempting to add the following line as 

the f or’s suite, please don ? t do this: 

return len (resp.content), resp.status code, resp.uri 



When a function executes a return statement, the function terminates. You don’t want 
this to happen here, as the gen_f rom_urls function is being called as part of a 
for loop, which is expecting a different tuple of results each time the functiones called. 

But, if you can’t execute return, what are you to do? 


Use yield instead. The yield keyword was added to Python to support the 
creation of generator functions, and you can use it anywhere a return is used. 
When you do, your function morphs into a generator function that can be “called” 
from any iterator, which, in this case, is from within your for loop: 

import requests 

def gen_from_urls(uris: tuple) -> tuple: 

for resp in (requests.get(uri) for uri in uris): 

yield len(resp.content), resp.status code, resp.uri 


Wse "yield" to «U 
each line ot vesults 
ihc 6\£T 

response io the waitina 
■for' loop. Ren.en.ber: 
t>0H’T use "return” 



Let’s take a closer look at what 5 s going on here. 
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generator functions at work 


Traciwg Your Generator Function, 1 of 2 


To understand what happens when your generator function runs, let’s trace the 
execution of the following code: 


|r*poY~t 70UV 



from uri utils import gen from uris 



uris = ('http://talkpython.fm', 'http://pythonpodcast.com'. 


De-Pine a 

■feuple ot 

URLs. 

'http://python.org') 


for resp len, status, uri, in gen from_urls(uris): 
print(resp len, '->', status, '->', uri) 


Use your genev-a-tov 

WW as pav~t a 

w -fo*r W loop. 


The first two lines of code are simple enough: the function is imported, and a tuple of 
URLs is defined. 

The fun starts on the next line of code, when the gen_f rom_urls generator 
function is invoked. Let 5 s refer to this for loop as “the calling code”: 


for resp len, status, uri, in gen from uris(uris): 



The interpreter jumps to the gen_f rom_urls function, and starts to execute its 
code. The tuple of URLs is copied into the function’s sole argument, and then the 
generator function’s for loop executes: 


The Mailing todes w -Por 

loop dommuhidates 
with the generator 
-funeti ok/s u -for w loop. 


def gen_from_urls(uris: tuple) -> tuple: 

for resp in (requests.get(uri) for uri in uris) 
yield len(resp.content), resp.status code. 


resp.uri 



The for loop contains the generator, which takes the first URL in the uris tuple and 
sends a GET request to the identifled server. When the HTTP response is returned 
from the server, the yield statement executes. 

This is where things get interesting (or weird, depending on your point of view). 

Rather than exeeuting, then moving on to the next URL in the uris tuple (i.e., 
continuing with the next iteration of gen_from_urls’s for loop), yield passes 
its three pieces of data back to the calling code. Rather than terminating, the gen_ 
f rom_urls function generator now waits, as if in suspended animation... 
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Traciwg Your Generator Function, 2 of 2 

When the data (as passed back by yield) arrives at the calling code, the for loop’s 
suite executes. As the suite contains a single call to the print BIF, that line of code 
executes and displays the results from the first URL on screen: 


print(resp_len, '->', status, uri) 

34591 -> 200 -> https://talkpython.fm/ 


The calling code’s for loop then iterates, calling gen_from_urls again...sort of 

This is almost what happens. What actually happens is that gen_f rom_urls is 
awakened from its suspended animation, then continues to run. The for loop within 
gen_f rom_urls iterates, takes the next URL from the uris tuple, and contacts the 
server associated with the URL. When the HTTP response is returned from the server, 
the yield statement executes, passing its three pieces of data back to the calling code 
(which the function accesses via the resp object): 

yield len (resp.content), resp.status code, resp.uri 


The three yielded 
piedes of data are 
takeh from -the "resp" 
objed-t returned 
by the Ve^uests" 
librarys "get" method- 


As before, rather than terminating, the gen_f rom_urls generator function now 
waits once more, as if in suspended animation... 

When the data (as passed back by yield) arrives at the calling code, the for loop’s 
suite executes print once more, displaying the second set of results on screen: 


34591 -> 200 -> https://talkpython.fm/ 
19468 -> 200 -> http://pythonpodcast.com/ 


The calling code’s for loop iterates, “calling” gen_from_urls once more, which 
results in your generator function awakening again. The yield statement is executed, 
results are returned to the calling code, and the display updates again: 


34591 

-> 

200 

-> 

19468 

-> 

200 

-> 

47413 

-> 

200 

-> 


https://talkpython.fm/ 
http://pythonpodcast.com/ 
https://www.python.org/ 


At this point, youVe exhausted your tuple of URLs, so the generator function and the 
calling code 5 s for loop both terminate. It 5 s as if the two pieces of code were taking 
turns to execute, passing data between themselves on each turn. 

Let 5 s see this in action at the >>> prompt. It’s now time for one last Test Drive. 
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don’t be sad 



In this, the last TestDrive in this book, let’s take your generator function for a spin. As has been our 
practice ali along, load your code into an IDLE edit window, then press F5 to exercise the function at 
the >» prompt. Follow along with our session (below): 


tteves “the w 
v-owg_u\r Is 

genev-atov- -fundtion ^ 
'm the w u\rl_utiUpy 



# # # url utils.py -/Users/paul/Desktop/NewBook/ch12/url utils.py (3.5.2) 

import requests 

def gen_from_urls (uris: tuple) -> tuple : 

for resp in (requests.get(uri) for uri in uris): 

yield len (resp.content), resp.status code, resp.uri 


Ln: 8 Coi: 0 



The first example below shows gen_from_urls being called as part of a for loop. As expected, 
the output is the same as that obtained a few pages back. 

The second example below shows gen_f rom_urls being used as part of a dictcomp. Note how 
the new dictionary only needs to store the URL (as a key) and the size of the landing page (as the 
value). The HTTP status code is not needed in this example, so we teli the interpreter to ignore it 
using Python’s default variable name (which is a single underscore character): 


Python 3.5.2 Shell 


£aeh line o$ 

vesul-ts appeavs, 
a-ftev a shov-t 
pause, as -the data 
is genev-ated by 
the -function- 


This dieteomp 
assoeiates the 
MRL with the 
Icngth o-P its 
landing page. 



»> 

»> 


»> for resp_len, status, uri in gen_from_urls(uris) 
print (resp_len, status, '->', uri) 


31590 -> 200 -> http://headfirstlabs.com/ 
78722 -> 200 -> http://www.oreilly.com/ 
128244 -> 200 -> https://twitter.com/ 

»> 

»> urls_res = {uri: size for size, 

»> 

»> import pprint 
»> 

»> pprint .pprint (urls_res) 

{'http://headfirstlabs.com/': 31590 
'http://www.oreilly.com/': 78722, 

'https://twitter.com/': 128244} 

>» * 



Pass -the -twfle 

Jc URU -bo 

-the ynevabov 

■Vunt-tio»- 


uri in gen from uris(uris)} 


The underseoire telis the 
^ode to >gnome the yielded 
^"TP status codt value. 


Ln: 271 Coi: 0 
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Pvetty-fV-inting the Vljres W dittionavy eon-fivms 
that the genevatov -funetion ean be used within a 
dieteor^p (as well as within a w W ; loop). 

















advanced iteration 


Cowcluding Rcmarks 


The use of comprehensions and generator functions is often regarded as an advanced 
topic in the Python world. However, this is mainly due to the fact that these features 
are missing from other mainstream programming languages, which means that 
programmers moving to Python sometimes struggle with them (as they have no 
existing point of reference). 

That said, over at Head First Labs , the Python programming team loves comprehensions 
and generators, and believes that with repeated exposure, specifying the looping 
constructs that use them becomes second nature. They can’t imagine having to do 
without them. 


Even if you find the comprehension and generator syntax weird, our advice is to 
stick with them. Even if you dismiss the fact that they are more performant than the 
equivalent for loop, the fact that you can use comprehensions and generators in 
places where you cannot use a for loop is reason enough to take a serious look at 
these Python features. Over time, and as you become more familiar with their syntax, 
opportunities to exploit comprehensions and generators will present themselves as 
naturally as those that teli your programming brain to use a function here, a loop 
there, a class over here, and so on. Here’s a review of what you were introduced to in 
this chapter: 



BULLET POINTS 


■ When it comes to working with data in files, Python has 
options. As well as the Standard open BIF, you can 
use the facilities of the Standard library’s csv module to 
work with CSV-formatted data. 

■ Method chains allow you to perform Processing on 
data in one line of code. The string. strip () . 
split () Chain is seen a lot in Python code. 

■ Take care with how you order your method chains. 
Specifically, pay attention to the type of data returned 
from each method (and ensure type compatibility is 
maintained). 


the most popular variant “in the wild.” Seasoned Python 
programmers refer to these constructs as listcomps, 
dictcomps, and setcomps. 

A listcomp is code surrounded by square brackets, 
while a dictcomp is code surrounded by curly braces 
(with colon delimiters). Asetcomp is also code 
surrounded by curly braces (but without the dictcomp’s 
colon). 

There’s no such thing as a “tuple comprehension,” as 
tuples are immutable (so it makes no sense to try to 
dynamically create one). 


A for loop used to transform data from one format to 
another can be reworked as a comprehension. 

Comprehensions can be written to process existing lists 
dictionaries, and sets, with list comprehensions being 


If you spot comprehension code surrounded by 
parentheses, you’re looking at a generator (which can 
be turned into a function that itself uses yield to 
generate data as needed). 


As this chapter concludes (and, by definition, the core content of this book), we 
have one final question to ask you. Take a deep breath, then flip the page. 
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gotta love whitespace 


One Final Question 

OK. Here goes, our final question to you: at this stage in this book, doyou even notice 
Pythonis use of significant whitespace? 

The most common complaint heard from programmers new to Python is its use of 
whitespace to signify blocks of code (instead of, for instance, curly braces). But, after a 
while, your brain tends not to notice anymore. 

This is not an accident: Python’s use of significant whitespace was intentional on the 
part of the language 5 s creator. 

It was deliberately done this way, because code is read more than it ? s written. 

This means code that conforms to a consistent and well-known look and feel is easier 
to read. This also means that Python code written 10 years ago by a complete stranger 
is stili readable by you today because of Python’s use of whitespace. 

This is a big win for the Python community, which makes it a big win for you, too. 
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Chapter 12$ Code 


This is 'do__dohVc\rt.py w 


from datetime import datetime 
import pprint 

def convert2ampm(time24: str) -> str: 

return datetime.strptime(time24, '%H:%M').strftime('%I:%M%p') 

with open('buzzers.csv') as data: 
ignore = data.readline() 
flights = {} 
for line in data: 

k, v = line.strip() .split ( ', ') 
flights[k] = v 

pprint.pprint(flights) 
print() 

fts = {convert2ampm(k): v.title() for k, v in flights.items()} 

pprint.pprint(fts) 
print() 

when = {dest: [k for k, v in fts.items() if v == dest] for dest in set(fts.values())} 

pprint.pprint(when) 
print() 




import requests 

def gen_from_urls(uris: tuple) -> tuple: 

for resp in (requests.get(uri) for uri in uris): 

yield len (resp.content), resp.status code, resp.uri 
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so long (and thanks for all the fish) 


Ifs Time to &o... 



YouVe oh your way! 

We’re sad to see you leave, but nothing would make us happier than you taking what you’ve learned about Python in 
this book and putting it to use. You’re at the start of your Python journey, and there’s always more to learn. Of course, you’re 
not quite done with this book just yet. There’s the five (yes: five!) appendixes to work through. We promise they’re not that 
long, and are well worth the effort. And, of course, there’s the index—let’s not forget about the index! 

We hope you’ve had as much fun learning about Python as we’ve had writing this book for you. It’s been a blast. Enjoy! 
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appendix a: InstaUaffcn 


X Installing Python + 



First things first: let’s get Python installed on your computer. 

Whether youYe running on Windows, Mac OS X, or Linux, Python’s got you covered. How 
you install it on each of these platforms is specific to how things work on each of these 
operating systems (we know...a shocker, eh?), and the Python community works hard 
to provide installers that target all the popular systems. In this short appendix, you’ll be 
guided through installing Python on your computer. 


this is an appendix 
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works on Windows 


Iwstall Python $ ow Wiwdows 

Unless you (or someone else) has installed the Python interpreter onto your 
Windows PG, it is unlikely to be preinstalled. Even if it is, let’s install the 
latest and greatest ver sion of Python 3 into your Windows computer now. 

If you already have a version of Python 3 installed, it’ll be upgraded. If you 
have Python 2 installed, Python 3 will install alongside it (but won’t interfere 
with your Python 2 in any way). And if you don’t have any version of Python 
yet, well, you soon will! 


Pownload, then install 

Point your browser to www.python.org , and then click the Downloads tab. 

Two large buttons will appear, offering the choice of the latest version of 
Python 3 or Python 2. Click on the Python 3 button. Go ahead and save the 
file for download when prompted. After a little while, the download will 
complete. Locate the downloaded file in your Downloads folder (or wherever 
you saved it), then double-click on the file to start the install. 

A Standard Windows installation process begins. By and large, you can click on 
JVext at each of the prompts, except for this one (shown below), where you’11 
want to pause to make a configuration change to ensure Add Python 3.5 to Path 
is selected; this ensures Windows can find the interpreter whenever it needs to: 


No*tc : As *tv%is book huvtles 
towavd 'rb date with 
the pvintin^ fv-css, the 
ne%*t vevsion ot Python 'i 
(velease is due out- A s 
this wont be until the end 
Jc 2-0\l> (a meve handtul ot 
v/eeks *attev* this book 
publishes), weVe showin$ 3.5 
in these stveenshots. Pon t 
Y/oVV*y about roat^hin^ the 
vevsion we have heve- ^o 
ahead and download/install 



you Ve installin^ is 
likely dittevent 
-(Vow* this. Pon t 
wovvy, youvs is the 
latest vevsion and 
installs in the sa^e 
way. 


the latest- 


Python 3.5.2 (64-bit) Setup 




python 

for 

Windows 


Install Python 3.5.2 (64-bit) 

Select Install Now to install Python with default settings, or choose 
Customize to enable or disable features. 


-► Install Now 

C:\Users\Paul Bariy\AppData\LocarvPrograms\Python\Python35 

Includes IOLE, pip and documentation 
Creates shortcuts and file associations 


Customize installation 

Choose location and features 




This is v-eally 
ir*po\rtant : 
be suve to 
tuvn on this 
option beto ve 
elidking on 
"Install /W' 
at this dialog. 
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Check Python $ on Windows 

Now that the Python interpreter is installed on your Windows machine, let’s run a 
few checks to confirm all is OK. 

For starters, you should have a new group on your Start menu under All Programs. 
WeVe included what it looks like on one of the Head First Labs 3 Windows 7 
machines. Yours should look similar. If it doesn’t, you may need to redo the 
installation. Windows 8 users (or higher) should also have a new group similar to 
this. 


Let 5 s examine the items in the Python 3.5 group from 
the bottom up. 

The Python 3.5 Modules Docs option provides access 
to all of the documentation included with all of the 
installed modules that are available within your Python 
system. Youil be learning lots about modules as you 
work through this book, so you don’t need to worry 
about doing anything with this option right now. 


The Python 
mstallev adds a 
yro to 

All PvoyamS list- 


The Python 3.5 Manuals option opens the entire set of Python language 
documentation in the Standard Windows help utility. This material is a copy of the 
Python 3 documentation available on the Web. 



The Python 3.5 option fires up a text-based interactive command prompt, >>>, which 
is used to experiment with code as you write it. We’ll have more to say about the >>> 
prompt starting from Ghapter 1. If you have clicked on this option to try it out and 
are now at a loss as to what to do, type quit () to escape back to Windows. 

The final option, IDLE (Python 3.5), runs the Python integrated development 
environment, which is called IDLE. This is a very simple IDE that provides access to 
Python’s >>> prompt, a passable text editor, the Python debugger, and the Python 
documentation. Weil be using IDLE a lot in this book, starting in Ghapter 1. 


lt'$ Python $ on Windows, sort of... 

Python’s heritage is on Unix and Unix-like Systems, and this can sometimes come 
through when you’re working in Windows. For instance, some Software that is 
assumed to exist by Python isn’t always available by default on Windows , so to get the 
most out of Python, programmers on Windows often have to install a few extra bits 
and pieces. Let’s take a moment to install one such bonus piece to demonstrate how 
these missing bits can be added when needed. 
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Windows need pyreadline 


Add to Python i on Windows 


Sometimes programmers using the Windows version of Python feel like they 
are being short-changed: some of the features assumed (by Python) on those 
other platforms are “missing” from Windows. 

Thankfully, some enterprising programmers have written third-party modules 
that can be installed into Python, thus providing the missing functionality. 
Installing any of these modules involves only a little bit of work at the 
Windows command prompt. 

As an example, let’s add Python’s implementation of the popular re adi i ne 
functionality to your Windows version of Python. The pyreadline module 
provides a Python version of readline, effectively plugging this particular 
hole in any default Windows installation. 

Open up a Windows command prompt and follow along. Here, we’re going 
to use a Software installation tool (included in Python 3.5) to install the 
pyreadline module. The tool is called pip, short for “Python Index 
Project,” named after the work that spawned pip’s creation. 



The readline library 
implements a set of functions 
that provide interactive text- 
editing facilities (typically 
at command lines). The 
pyreadline module 
provides a Python interface to 
readline. 


At the Windows command prompt, type pip Install pyreadline: 



File Edit Window Help InstallingPyReadLine 


Microsoft Windows [Version 6.1.7601] 

Copyright (c) 2009 Microsoft Corporation. All r 
C:\Users\Head First> > 

C:\Users\Head First> pip install pyreadline 
Downloading/unpacking pyreadline 


'fito the promp-fc. 

I 


hts reserved 


Successfully installed pyreadline 
Cleaning up. . . 

C:\Users\Head First> 


Youll see lo-ts ot 
messa^es here- 


It you see -this messaqe. 
all is 0K. 3 



A/iaW^ suve 'yow are 


And with that, pyreadline is installed and ready to go on Windows. 

You can now flip back to Ghapter 1 to get started with some sample Python 
code. 
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Install Python 3 oh Mac OS X (wacOS) 

Python 2 comes preinstalled on Mac OS X by default. But this is no use to us, 
as we want to use Python 3 instead. Thankfully, when you visit the Python 
website ( http://www.python.org ), it is smart enough to work out that you’re 
using a Mac. Hover your mouse over the Download tab, then click the 3.5.x 
button to download the Mac installer for Python. Select the latest version of 
Python 3, download its package, and then install in the usual “Mac way.” 


f\ s-tandavd Mat 
Qg y, mstallaW 
Dy-oftva*» vov 
python W2- and 

a bove- l£ y ' ave 

a move vetent 
velease ■tha*' ^at 
vieVe sWtn<\ 

Vieve, tKatsVme- 

mstall a «vf. 




Jus*t kccy clickmj 
until mstallcd- 


Using a package wawager 

On Macs, it is also possible to use one of the popular open source package 
managers , namely Homebrew or MacPorts. If you have never used either of 
these package managers, feel free to skip this little section and jump over to 
the top of the next page. If, however, you are already using either of these 
package managers, here are the commands you need to install Python 3 on 
your Mac from inside a terminal window: 

• On Homebrew, type brew install python3. 

• On MacPorts type port install python3. 

And that’s it: you’re golden. Python 3 is ready for action on Mac OS X —let 5 s 
take a look at what gets installed. 
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set your path 


Check and Configure Python i on Mac OS X 

To see if the install succeeded on Mac OS X, click on the Applications icon on your 
dock, then look for the Python 3 folder. 

Click on the Python 3 folder and you’11 see a bunch of icons (below). 


The Python $ folder on Mac OS X 

The first option, IDLE : is by far the most useful, 
and it is how you will interact with Python 3 most 
of the time while learning the language. Choosing 
this option opens Python’s integrated development 
environment called IDLE. This is a very simple IDE 
that provides access to Python’s >>> interactive 
prompt, a passable text editor, the Python debugger, 
and the Python documentation. We’ll be using IDLE 
a lot in this book. 

The Python Documentation.html option opens a local 
copy of Python’s entire documentation in HTML 
within your default browser (without requiring you to 
be Online). 

The Python Launcher option is automatically run 
by Mac OS X whenever you double-click on an 
executable file containing Python code. Although this 
may be useful for some, at Head First Labs we rarely 
use it, but it’s stili nice to know it’s there if we ever do 
need it. 

The last option, Update Shell Profile.command , updates 
the configuration files on Mac OS X to ensure the 
location of the Python interpreter and its associated 
Utilities are correctly added to your operating systenTs 
path. You can click on this option now to run this 
command, then forget about ever having to run it 
again—once is enough. 

YouYc ready to run on Mac OS X 

And with that, you’re all set on Mac OS X. 

You can now skip back to Chapter 1 and get started. 


"The Pytho* 3 ^oldev* 
-(Vor* v/ithin the 
Applications -Poldev* on 
Mac X 


N/ 




IDLE 


Python 3.5 


License.rtf 


HTML 

Python D... tion.html 



Python Launcher 



ReadMe.rtf Update S...ommand Open in Finder 




I* the Pock, click the 
Applications iCon, and 
then select the Python 
3 £oldev. 
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appendix a: installation 


Install Python $ on Linux 

If you are running a recent distributiori of your favorite Linux , the really great 
news is that you most likely have Python 2 and Python 3 already installed. 

Here i a quick way to ask the Python interpreter to fess up its currently 
installed version number; open up a command line and type: 


Be £a\re-Pu|: -fchats 

$ python3 -V ~~ WPP BRCA&B 

^ u u 

3.5.2 v • 

Hov/ too\ is 
Ouv* Linu* hds Lhc 
latcs-t Python 3 
installed- 

If, after you issue this command, Linux complains that it can’t find python3, 
you need to install a copy. How you do this depends on the Linux distribution 
you are running. 

If your Linux is one based on the popular Debian or Ubuntu distribution (as is 
the one we use at Head First Labs), you can use the apt-get utility to install 
Python 3. Here i the command to use: 



$ sudo apt-get install python3 idle3 


If you are running ajwm-based or rpm- based distribution, use the equivalent 
command for those Systems. Or fire up your favorite Linux GUI and use your 
distributioni GUI-based package manager to select python 3 and i die 3 
for installation. On many Linux Systems, the Synaptic Package Manager is a 
popular choice here, as are any number of GUI-based Software installers. 

After installing Python 3, use the command from the top of this page to check 
that all is OK. 

No matter which distribution you use, the python3 command gives you 
access to the Python interpreter at the command line, whereas the i die 3 
command gives you access to the GUI-based integrated development 
environment called IDLE. This is a very simple IDE that provides access to 
Pythoni >>> interactive prompt, a passable text editor, the Python debugger, 
and the Python documentation. 

We’ll be using the >>> prompt and IDLE a lot in this book, starting in 
Ghapter 1, which you can flip back to now. 


Be sure to select 
tlie "python3” and 
"idle3” packages 
lor installation on 
Linux. 


you are here ► 
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X Deploying Your Webapp + 



At the end of Chapter 5, we claimed that deploying your webapp 
to the cloud was only 10 minutes away. 

It’s now time to make good on that promise. In this appendix, we are going to take you through 
the process of deploying your webapp on PythonAnywhere, going from zero to deployed in 
about 10 minutes. PythonAnywhere is a favorite among the Python programming community, 
and it’s not hard to see why: it works exactly as you’d expect it to, has great support for Python 
(and Flask), and—best of all—you can get started hosting your webapp at no cost. Let’s check 
out PythonAnywhere. 


this is an appendix 
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getting ready 


Stcp 0: A Uttle Prep 

At the moment, you have your webapp code on your computer in a folder 
called webapp, which contains the vsearch4web . py file and the static 
and templa te s folders (as shown below). To prepare all this stuff for 
deployment, create a ZIP archive file of everything in your webapp folder, 
and call the archive file webapp . zip: 



rn-to an avdWive 
dalled “webapp-»? • 



vsearch4web.py 



hf.css 



In addition to webapp . zip, you also need to upload and install the 
vsearch module from Ghapter 4. For now, all you need to do is locate the 
distribution file that you created back then. On our computer, the archive 
file is called vsearch-1.0 . tar . gz and it’s stored in our mymodules/ 
vsearch/di st folder (on Windows , the file is likely called vsearch- 
1.0. zip). 

You don’t need to do anything with either archive file right now. Just make a 
note of where both archive files are on your computer so that they are easy to 
find when you upload them to PythonAnywhere. Feel free to grab a pencil and 
scribble down each archive file’s location here: 



ftedall tvon» Chap-tev- ‘S 
■fcha-t Pyihon's "setup-tools 

module dvea-tes Zl Ps on 
Windows, and .-fcan.gz. -files 
on evev-ything else- 


webapp.zip 


vsearch-1.0.tar .gz 



This is "vsean-dh.zjp" instead i-P 

you \rc oh l^ihdows. 
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appendix b: pythonanywhere 


Stcp 1: Sigw Up for PythowAwy whcrc 

This step couldn’t be any easier. Surf over to pythonanywhere.com, then click on 

the Pricing & signup link: 



S-tart Kcvc- 


Glick on the big, blue button to create a Beginner account , then fili in the details on the signup form: 


D 


fi pythonanywhere.com 




y 

rgfej pythonanywhere 

Plans and pricing 

Beginner: Free! 


Send feedback Forums Help Blog Pricing & signup Log in 




A limited account with one web app at your-username.pythonanywhere.com, restricted outbound Internet access from your 

apps, low CPU/bandwidth, no IPython notebook support. 

It works and 1 *'°- ' ' — ■*«««*' 


Create a Beginner account 




AII of our paid pians come with a no-qi tble 30-day money-back guarantee — you're billed monthly and you can 

contract length is just one month. You ge unrestricted Internet access from your applications, unlimited in-browser P} 
and full SSH access to your account. AII ccounts (including free ones) have screen-sharing with other PythonAnywf 
(though you'll need to get a certificate fo/your own domains). 



This is “the 
w W si$»uy ’ oy-tion. 


D 


A pythonanywhere.com 


□' ■ 


anywhere 




Send feedback Forums Help Blog Pricing & signup Log in 


Create your account 


Username: 

Email: 

Password: 
Password (again): 


headfirst2e 


headfirst2e@gmail.com 



c 



\ 



Fili ih this -f< 


o\rr*. 


Register 


We promise not to spam or pass your details on to anyone else. 


Copyright © 2015 PythonAnywhere LLP — Terms — Privacy 
“Python" is a registered trademark of the Python Software Foundation. 


If all is well, the PythonAnywhere dashboard appears. Note: you are both registered and signed in at this point: 


MI < > , ED 

pythonanywhere 


fi pythonanywhere.com 


c 


ib 


1± 


Send feedback Forums Help Blog Dashboard Account Log out 



Pyt 

dasKboard Noie 
-t.be -fwe "tabs 
available -to y 01 *- 


Consoles Files Web Schedule Databases 

Thank you! 

You're now signed up and logged in to your PythonAnywhere account: headfirst2e. 

We've sent an email to headfirst2e@gmail.com. If you click the link in the email to confirm your email address, we'll be able to reset your password if 
you forget it in the future. 

What next? 

We've got some helpers to get you started with some common tasks — why not try one out? Alternatively, if you don't want any help, just click the “X" 
button above and to the right. 

• I want to start learning Python 

• I want to follow the Django Tutorial 

• I want to create a web application 

• I have built a web app on my local PC and want to deploy it on PythonAnywhere 

• I want to clone and hack on my GitHub project 

• I want to check out the PythonAnywhere Education features 

If there's something else you think we should have here, click here to let us know. 
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upload your code 


Step 2 : Upload Your Files to the Cloud 

Glick on the Files tab to view the folders and hies available to you: 


C\\tV hev-e* 


Here 
a\rc your 
-folders. 


• • < | > i E2_ _ 

anywhere 

Consolem/ Files M/eb 


fii pythonanywhere.com 


± g [V 


e ¥ 


Send teedback Forums Help Blog Dashboard Account Log out 



Schedule Databases 



Tome > a headfirst2e 


Enter new directory name New 


ID Open Bash console here 


Enter new file name 


New 


i mysite/ 


Upload a file: choose File no file selected 


Ii .bashrc i B 

Ii .giteonfig A B 

Ii .profile i. B 

Ii .pythonstartup.py 1 B 

Ii .vimrc A B 

Ii README.txt 


2016-01-01 13:61 659bytes 
2016-01-01 13:51 266bytes 
2016-01-01 13:51 79 bytes 
2016-01-01 13:51 77 bytes 
2016-01-01 13:51 4.6 KB 


A B'fi 2016-01-01 13:51 235 bytes 

Copyright © 2015 PythonAnywhere LLP — Terms — Privacy 
“Python" is a registered trademark of the Python Software Foundation. 



Use the Upload a file option to locate and upload the two archive hies from Step 0: 


Here are ihe 
-Piles ih your 

-Poldev. 


Wse this 
optioh -to 

upload eacM 
ot your 
a\r£hive -Piles. 




<_ >1 J^L 

cy thonany where 

Consoles Files Web 

/ > home > fi headfirst2e 

Enter new directory name New 

fi .virtualenvs/ fi 

S mysite/ fi 



fi pythonanywhere.com 


di g □! 


Send feedback Forums Help Blog Dashboard Account Log out 


Schedule Databases 


H Open Bash console here 


Enter new filo n 


New 



Ii .bashrc A B 

Ii giteonfig A B 

Ii .profile A B 

Ii .pythonstartup.py A B 

Ii .vimrc A B 

Ii README.txt 
k vsearch-1.0.tar.gz 
k webapp.zip 


2016-01-01 13:61 559 bytes 
2016-01-01 13:51 266 bytes 
2016-01-01 13:51 79 bytes 
2016-01-01 13:51 77 bytes 
2016-01-01 13:51 4.6 KB 


A B fi 2016-01-01 13:51 235 bytes 
A fi 2016-01-01 13:58 899 bytes 
A fi 2016-01-01 13:63 3.8 KB 

Copyright © 2015 PythonAnywhere LLP — Terms — Privacy 
"Python" is a registered trademark of the Python Software Foundation. 


? 


V°vVe going io 

tlitk -fchis Jj n k 

■fi a momerrt- 


Me* you re done 
uploadiha, bo-fch 
amhive -Pi| cs 
should appear oh 

“the lis-t o-p -piles ih 
y°ur home -Polder. 


You’re now ready to extract and install these two uploaded archive hies, and you’11 do that during 
Step 3. To get ready, click the Open a bash console here link at the top right of the above page. This 
opens up a terminal window in your browser window (on PythonAnywhere). 


532 Appendix B 





















appendix b: pythonanywhere 


Stcp $: Extract and Install Your Code 

When you click the Open a bash console here link, PythonAnywhere responds by replacing the Files 
dashboard with a browser-based Linux console (command prompt). You’re going to issue a few 
commands to extract and install the vsearch module as well as your webapp’s code within 
this console. Begin by installing vsearch into Python as a “private module” (i.e., just for your 
use) using this command (be sure to use vsearch-1.0 . zip if you’re on Windows ): 

python3 -m pip install vsearch-1.0.tar.gz —user 


R \ ay \ -the 

£.omm3nd* 

Su ££ess/ 


on 


fl pythonanywhere.com 


± fl Gl 


S ¥ 


honanywhere 

Bash console 2149185 


Send feedback Forums Help Blog Dashboard Account Log out 

'X Share with others 


13:59 ~ $ Is a 

README.txt mysite vsearch-1.0.tar.gz webapp.zip 
14:00 ~ $ python3 -m pip install vsearch-1.0.tar.gz —user 
Processing ./vsearch-1.0.tar.gz 
Building wheels for collected packages: vsearch 
Running setup.py bdist_wheel for vsearch 

Stored in directory: /home/headfirst2e/.cache/pip/wheels/eb/fc/ad/734c9fbb2d6cfc97f96f045b42e63d536eb2alf47 
191735fce 

5uccessfully built vsearch 
Installing collected packages: vsearch 
Successfully installed vsearch 
14:00 ~ $ 


Copyright © 2015 PythonAnywhere LLP — Terms — Privacy 
"Python" is a registered trademark of the Python Software Foundation. 


W )) 

USCV* ChSUVCS 

the Veav-eh” 
module is ihs-tallcd 

-Poir youir use oh ly. 

Pythoh^hywhere 

docs hot allov/ you 
to ihs^lf 3 module 
-for eve\ryohe ; s use 

0«t y° u,r owh). 


With the vsearch module successfully installed, it 5 s time to turn your attention to your 
webapp 5 s code, which has to be installed into the mysite folder (which already exists on your 
PythonAnywhere horne folder). To do this, you need to issue two commands: 

un zip webapp.zip 
mv webapp/* mysite 


Uh\>atk youv 

eba^s 


w 
eode- 


y 

pythonanywhere 

Send feedback Forums Help Blog Dashboard Account Log out 

Bash console 2149185 

tX Share with others 


Yoia should 
see messages 
similav -to 
these. 


r 




D 


fi pythonanywhere.com 


di fl yz 


Building wheels for collected packages: vsearch • 

Running setup.py bdist_wheel for vsearch 

Stored in directory: /home/headfirst2e/.cache/pip/wheels/eb/fc/ad/734c9fbb2d6cfc97f96f045b42e63d536eb2alf47 
191735fce 

Successfully built vsearch 
Installing collected packages: vsearch 
Successfully installed vsearch 
14:00 ~ $ unzip webapp.zip 
Archive: webapp.zip 
creating: webapp/ 
inflating: webapp/.DS_Store 

creating: _MACOSX/ 

creating: _MACOSX/webapp/ 

inflating: _MACOSX/webapp/.DS_Store 

creating: webapp/static/ 
inflating: webapp/static/hf.ess 
creating: webapp/templates/ 
inflating: webapp/templates/base.html 
inflating: webapp/templates/entry.html 
inflating: webapp/templates/results.html 
inflating: webapp/vsearch4web.py 
14:08 ~ $ mv webapp/* mysite/ 

14:08 ~ $ 


Copyright © 2015 PythonAnywhere LLP — Terms — Privacy 
“Python" is a registered trademark of the Python Software Foundation. 


•••theh move 
■fche todt ih-fco 
“the mysi-fce” 
tolder. 


you are here ► 
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flask on pythonanywhere 


Step 4: Create a Starter Webapp, 1 of 2 

With Step 3 done, return to the PythonAnywhere dashboard and select the Web tab, where 
PythonAnywhere invites you to create a new starter webapp. You’11 do this, then swap out the 
starter’s webapp code for your own. Note that each Beginner account gets one webapp for 
free; if you want more, you’11 have to upgrade to a paid account. Luckily—for now—you 
only need the one, so let’s keep going by clicking a new web app\ 



As you are using a free account, your webapp is going to run on the site name shown on the 
next screen. Glick the JVext button to proceed with PythonAnywhere’s suggested site name: 




E 


anywhere 


fi pythonanywhere.com 


6 


(h Qg 


Send feedback Forums Help Blog Dashboard Account Log out 


Consoles Fi 


o 


Add a new web app 



Create new web app 

Your web app's domain name 

Your account doesn't support custom domain names, so your PythonAnywhere web app will live at 
headfirst2e.pythonanywhere.com. 

Want to change that? Upgrade now! 

Otherwise, just click "Next" to continue. 


PythonAnywhev-e 
1'is-b your si*te s 


C\\cM "tbis “to 

Cancel «Back Next» 

- A 



Glick Next to continue with this step. 
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appendix b: pythonanywhere 


Stcp 4: Create a Starter Webapp, 2 of 2 

PythonAnywhere supports more than one Python web framework, so the next screen offers 
you a choice among the many supported Systems. Pick Flask, then select the version of 
Flask and Python you wish to deploy to. As of this writing, Python 3.4 and Flask 0.10.1 
are the most up-to-date versions supported by PythonAnywhere , so go with that combination 
unless a newer combination is offered (in which case, pick the newer one instead): 


• • < 


BD 


fi pythonanywhere.com 


di O ■ 


%T: 


anywhere 

Create new web app 


Send feedback Forums Help Blog Dashboard Account Log out 


Consoles Fi 

Selert a Python Web framework 

O Add a new web app ...c select "Mar.. 'I configuration" if you want detailed control. 

» Django 

» Bottle 

> Manual configuration (including virtualenvs) 

What other frameworks should we have here? Send us some feedback using the lipk at the top of 
the page! •(')#< [□ 



Cancel 




Consoles Fi 

O Add a new web app 


anywhere 

Create new web app 


Selec-t "Flask" &*- your 
web *PP< then dhoose ihe 
£ost up-to-date Python/ 
rlask dor*bih^tioh. 


Select 



fli g 


S er/i feedback Forums Help Blog Dashboard Account Log out 


Python 3.4 (Flask 0.10.1) 

» Pytnon j.o ^riciai\ \j. i o. I J 
» Python 2.7 (Flask 0.9) 

Note: If you'd like to use a different version of Flask to the default version, you can use a 
virtualenv for your web app. There are instructions here. 


Yoffre nearly there. The next screen 
offers to create a quickstart Flask webapp. 
Go ahead and do that now by accepting 
the values on this page and clicking on 
the Next button to continue: 


Cancel 

« Back 

Next » 

J. 



• • • < 


m 


fi pythonanywhere.com 


Cj 


& 




+ 


%T: 


anywhere 

Create new web app 


Send feedback Forums Help Blog Dashboard Account Log out 


Consoles Fi 

Quickstart new Flask project 

O Add a new web app Enter a path for a Python file you wish to use to hold your Flask app. 

If this file already exists, its contents will be overwritten with the 
new app. 

Path 

/home/headfirst2e/mysite/flask_app.| 



Cl itk here. 




Cancel 

« Back 

Next » 

A 


don-t need io 

ilia l w e *r w As 

sooh as you dhoose 
the dorhbihd-tioh you 
-this sdree* 
appeav-s. 
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import your module 


Stcp 5 : Cowfigure Your Wcbapp 

With Step 4 complete, you are presented with the Web dashboard. Don’t be tempted to click that big, 
green button just yet—you haven’t told PythonAnywhere about your code yet, so hold off on running 
anything for now. Instead, click in the long link to the right of the WSGI configuration file label: 


E 


fi pythonanywhere.com 


fi 


fi) 


+ 



✓ headfirst2e.pythonanywhere.com Configuration for headfirst2e.pythonanywhere.com 

It’s temftmfy but 

O Add a new web app 

Reload: 



O Reload 


POlN 1 CUCK. 


headfirst2e. pythonanywhere.com 


Traffic: 

How busy is your site? 

This month (previous month) 0 (0) 

Today (yesterday) 0 (0) 

Hour (previous hour) 0 (0) 

Want some more data? Paying accounts get pretty charts ;-) 

Code: 

What your site is running. 

Source code: /home/headfirst2e/mysite 
WSGI configuration file: /var/www/headfirst2e_pythonanywhere_com_wsgi.py 

Python version: 3.4 

Virtualenv: 

Use a virtualenv to get different versions of flask, django etc from our default 


Clidk this |j h |< 

'ihsicad- 



Glicking that long link loads your newly created Flask webapp’s configuration file into PythonAnywhere^ 
web-based text editor. At the end of Ghapter 5, we told you that PythonAnywhere imports your webapp 
code before invoking app . run () for you. This is the file that supports that behavior. However, it 
needs to be told to reference your code, not the code in the starter app, so you need to edit the last line 
of this file fas shown belowh and then click Save : 


• • • < 


E 


fi pythonanywhere.com 




anywhere 


/> var > www > headfirst2e_pythonanywhere_com_wsgi.py 


Send feedback 


Keyboarc 


# This file contains the WSGI configuration required to serve up yc 

# web application at http://<your-username>. pythonanywhere.com/ 

# It works by setting the variable 'application' to a WSGI handler 

# description. 

# 

# The below has been auto-generated for your Flask project 
import sys 

# add your project directory to the sys.path 

project_home = u'/home/headfirst2e/mysite' 


13 

14 

15 
• 16 

17 


sys.path = [project_home] + sys.path 

# imiyj-' 1 — ri s^k app but need to call it "application" for WSGI to wc 
frcfi flask_appXmport app as application 


< 


E 


fi pythonanywhere.com 


db 


fi) 


+ 




honanywhere 

/ > var > www > headfirst2e„pythonanywhere_com_wsgi.py * 


Send feedback Forums Help Blog Dashboard Account Log out 


Keyboard shortcuts: Normal 


Save as... 


# This file contains the WSGI configuration required to serve up your 

# web application at http://<your-username>. pythonanywhere.com/ 

# It works by setting the variable 'application' to a WSGI handler of some 

# description. 

# 

# The below has been auto-generated for your Flask project 


1 
2 

3 

4 

5 

6 

7 

8 import sys 

9 

10 # add your project directory to the sys.path 

11 project_home = u'/home/headfirst2e/mysite' 

12* if project_home not in sys.path: 

13 sys.path = [project_home] + sys.path 

14 

15 # V need to call it "application" for WSGI to work 

16 fri orj app as application 

17 


The 

"Save" 

butfcoh 


Copyright © 2015 PythonAnywhere LLP — Terms — Privacy 
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Charae -the last line o£ this -f ile ^ 
to ve-fevente your vsearth^web 
module* 
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Step 6: Take Your Cloud-Pased Webapp for a $pm! 

Be sure to save your changed configuration file, then return to the Web tab on the dashboard. It is 
now time to click on that big, tempting, green button. Go for it! 


on 


fi pythonanywhere.com 


□ 


0 


+ 


✓ headfirst 2 e.pyth 0 nanywhere.com Configuration for headfirst2e.pythonanywhere.com 


O Add a new web app 


Reload: 


O Reload 

headfirst2e.pythonanywhere.com 


Traffic: 

How busy is your site? 

This month (previous month) 0 (0) 

Today (yesterday) 0 (0) 

Hour (previous hour) 0 (0) 

Want some more data? Paying accounts get pretty charts ;-) 

Code: 

What your site is running. 

Source code: /home/headfirst2e/mysite 
WSGI configuration file: /var/www/headfirst2e pythonanywhere com wsgi.py 

Python version: 3.4 

Virtualenv: 




After but a brief moment, your webapp appears in your browser, and it works exactly as it did when 
you ran it locally, only now anybody with an Internet connection and a web browser can use it too: 


0 0 0 < > EO headfirst2e.pythonanywhere.com 

C di 0 

Bash console 2149185 : headfirst2e : PythonAny... Web app setup : headfirst2e : PythonAnywhere 

Welcome to search4letters on the web! 

+ 


Welcome to search4letters on the web! 

Use this form to submit a search request: 


Phrase: this is a test of vsearch4web.py running on PythonAnywhere 
Letters: aeiou 


When you're ready, click this button: 


Do it! 


N. ^ Yt looking good •£<*■ 
'hPut and o„+D,.i 


' h P u t 3hd ou-tpu-t in 
tbe dloud. 


Anyone can use this web 
address {o in-tev-ac-t with 
youv- webapp. 


i • • tD 

Bash console 2149185 : headfirst2e : PythonAny.. 


headfirst2e.pythonanywhere.com 
Web app setup : headfirst2e : PythonAnywhere 


di 0 


Here are your results: 


Here are your results: 

You submitted the following data: 


Phrase: ; this is a test of vsearch4web.py running on PythonAnywhere 
Letters: aeiou 


When "this is a test of vsearch4web.py running on PythonAnywhere" is search for 
"aeiou", the following results are returned: 

{'e', 'i', 'u', 'a', 'o'> 




+ 


And with that, you’re done. The webapp you developed in Ghapter 5 has been deployed to 
PythonAnywhere’s cloud (in less than 10 minutes). There’s lots more to PythonAnywhere than whafs shown 
in this short appendix, so feel free to explore and experiment. At some point, remember to return 
the PythonAnywhere dashboard and log out. Note that, despite your logging out, your webapp keeps 
running in the cloud until you teli it otherwise. That’s pretty cool, isn’t it? 
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It was never our intentiori to try to cover everything. 

This book’s goal was always to show you enough Python to get you up to speed as quickly 
as possible. There’s a lot more we could’ve covered, but didn’t. In this appendix, we 
discuss the top 10 things that—given another 600 pages or so—we would’ve eventually 
gotten around to. Not all of the 10 things will interest you, but quickly flip through them 
just in case weVe hit on your sweet spot, or provided an answer to that nagging question. 
AII the programming technologies in this appendix come baked in to Python and its 
interpreter. 
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legacy code woes 


1. What About Python 2? 


As of this book’s publication date (late 2016) there are two mainstream flavors of 
Python in widespread use. You already know quite a bit about Python 3, as that’s the 
flavor you Ve used throughout this book. 

All new language developments and enhancements are being applied to Python 3, 
which is on a 12- to 18-month minor release cycle. Release 3.6 is due before 2016 ends, 
and you can expect 3.7 to arrive late in 2017 or early in 2018. 


Python 2 has been “stuck” at release 2.7 for some time now. This has to do with the 
fact that the Python core developers (the people who guide the development of Python) 
decided that Python 3 was the future, and that Python 2 should quietly go away. There 
were solid technical reasons for this approach, but no one really expected things to take 
so long. After all, Python 3—the future of the language—first appeared in late 2008. 

An entire book could be written on what’s happened since late 2008 until now. Suffice 
it to say, Python 2 stubbornly refused to go away. There was (and stili is) a huge 
installed base of Python 2 code and developers, with some domains dragging their 
heels when it comes to upgrading. There’s a very simple reason for why this is: Python 
3 introduced a handful of enhancements that broke backward compatibility. Put 
another way: there’s lots of Python 2 code that will not run unchanged in Python 3 (even 
though, at a first glance, it can be hard to teli Python 2 code from Python 3 code). Also, 
many programmers simply believed Python 2 was “good enough,” and didn’t upgrade. 

Recently (over the last year), there 5 s been a sea change. The switching rate from 2 to 3 
appears to be increasing. Some very popular third-party modules have released Python 
3-compatible versions, and this is having a positive effect on Python 3 adoption. 
Additionally, the Python core developers keep adding extra goodness to Python 3, making 
it a more attractive programming language over time. The practice of “backporting” 
the cool new features from 3 to 2 has stopped with 2.7, and although bug and security 
fixes are stili being applied, the Python core developers have announced that this activity 
will stop in 2020. The clock is ticking for Python 2. 



O 



Here’s the common advice offered when you’re trying to decide whether 3 or 2 is right 
for you: 


If you Ve startingf a new project, use Python 3 . 


You need to resist the urge to create more legacy code in Python 2, especially if you’re 
starting with a blank slate. If you have to maintain some existing Python 2 code, 
what you know about 3 carries over: you’11 certainly be able to read the code and 
understand it (it 5 s stili Python, regardless of the major version number). If there are 
technical reasons why the code has to remain running in Python 2, then so be it. If, 
however, the code can be ported to Python 3 without too much fuss, then we believe 
the gain is worth the pain, as Python 3 is the better language, and is the future. 
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2 . Virtual Programming Environwents 


Let 5 s imagine you have two clients, one with Python code that relies on one 
version of a third-party module, and another that relies on a different version of 
the same third-party module for their code. And, of course, you’re the poor soul 
who has to maintain both projects’ code. 

Doing so on one computer can be problematic, as the Python interpreter doesn’t 
support the installation of different versions of third-party modules. 

That said, help is at hand thanks to Python’s notion of virtual environments. 


A virtual environment lets you create a new, clean Python environment 
within which you can run your code. You can install third-party modules into 
one virtual environment without impacting another, and you can have as many 
virtual environments as you like on your computer, switching between them by 
activating the one you want to work on. As each virtual environment can maintain 
its own copy of whatever third-party modules you wish to install, you can use two 
different virtual environments, one for each of your client projects discussed above. 

Before doing so, however, you have to make a choice: use the virtual environment 
technology, called venv, that ships with Python 3’s Standard library , or install the 
virtualenv module from PyPI (which does the same thing as venv, but has 
more bells and whistles). It’s best if you make an informed choice. 



To learn more about venv, check out its documentation page: 


https://docs.python.org/3/library/venv.html 


To find out what virtualenv offers over and above venv, start here: 


https://pypi.org/proj ect/virtualenv/ 


Whether you use virtual environments for your projects is a personal choice. Some 
programmers swear by them, refusing to write any Python code unless it’s within 
a virtual environment. This may be a bit of an extreme stance, but to each their 
own. 

We chose not to cover virtual environments in the main body of this book. We feel 
virtual environments are—if you need them—a total godsend, but we don’t yet 
believe every Python programmer needs to use one for everything they do. 

We recommend you slowly back away from people who say that you aren’t a 
proper Python programmer unless you use virtualenv. 


/\ll he had “to do was use 
a viv-tual eiftVirowe*t 


o 
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everything’s an object 


$. More ow Object Oriewtatiow 


If youVe read through this entire book, by now you’11 (hopefully) appreciate what’s 
meant by this phrase: “In Python, everything’s an object.” 

Python’s use of objects is great. It generally means that things work the way you 
expect them to. However, the fact that everything 5 s an object does not mean that 
everything has to belong to a class, especially when it comes to your code. 


In this book, we didn’t learn how to create our own class until we needed one in 
order to create a custom context manager. Even then, we only learned as much as 
was needed, and nothing more. If youVe come to Python from a programming 
language that insists all your code resides in a class (with Java being the classic 
example), the way weVe gone about things in this book may be disconcerting. 
Don’t let this worry you, as Python is much less striet than Java (for instance) when 
it comes to how you go about writing your programs. 

If you decide to create a bunch of functions to do the work you need to do, then 
have at it. If your brain thinks in a more functional way, Python can help here 
too with the comprehension syntax, tipping its hat to the world of functional 
programming. And if you can’t get away from the fact that your code needs to 
reside in a class, Python has full-featured object-oriented-programming syntax 
built right in. 



O 


If you do end up spending a lot of time creating classes, check out the following: 

• @ staticmethod: A decorator that lets you create a static function 
within a class (which does not receive self as its first argument). 

• @classmethod: A decorator that lets you create a class method that 
expects a class as its first object (usually referred to as cis), not self. 

• @property:A decorator that allows you to redesignate and use a 
method as if it were an attribute. 

• _ slots _: A class directive that (when used) can greatly improve 

the memory efficiency of the objects created from your class (at the 
expense of some flexibility). 

To learn more about any of these, consuit the Python does ( https://docs.python . 
org/3/). Or check out some of our favorite Python books (discussed in the next 
appendix). 
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4. Formats for Striwgs and the Like 

The recurring example applicatiori used in this book displayed its output in a web 
browser. This allowed us to defer any output formatting to HTML (specifically, we 
used th zjinja2 module included with Flash). In doing so, we sidestepped one area 
where Python shines: text-based string formatting 

Let 5 s say you have a string that needs to contain values that won’t be known until 
your code runs. You want to create a message (msg) that contains the values so 
you can perform some later processing (perhaps you’re going to print the message 
on screen, include the message within an HTML page you’re creating with Jinja2 : 
or tweet the message to your 3 million followers). The values your code generates 
at runtime are in two variables: price (the price of the item in question) and 
tag (a catchy marketing tagline). You have a few options here: 

• Build the message you need using concatenation (the + operator). 

• Use old-style string formats (using the % syntax). 

• Take advantage of every string 5 s format method to build your message. 

Here 5 s a short >>> session showing each of these techniques in action (bearing in 
mind that you, having worked through this book, already concur with what the 
generated message is telling you): 


Python 3.5.2 Shell 



Which of these techniques you use is a personal preference, although there 5 s a 
bit of a push on to encourage the use of the format method over the other two 
(see PEP 3101 at https://www.py thon.org/dev /peps/pep-3101 /). You’11 find code 
in the wild that uses one technique over the other, and sometimes (and not at all 
helpfully) mixes all three. To learn more, start here: 

https://docs.python, org/3/library/string. html#formatspec 
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the sorted BIF rocks 


5 . ftcttiwg Thiwgs Sorted 

Python has wonderful built-in sorting capabilities. Some of the built-in data 
structures (lists, for example) contain sort methods that can be used to perform 
in-place ordering of your data. However, it is the sorted BIF that makes Python 
truly special (as this BIF works with any of the built-in data structures). 

In the IDLE session below, we first define a small dictionary (product), which 
we then process with a succession of f or loops. The sorted BIF is exploited to 
control the order in which each for loop receives the dictionary’s data. Follow 
along on your computer while you read the annotations: 


BIF is short 

lor "kuitt-in 
lunction.” 


pywt 
Out t-hc 

OK s extt*- 



The vav/ data 
looks sovted by 
value, but isr/t 


v-cally (this is move 
ludk tha^ anythih^ 
else). 


The output is 
ovdeved -Cvom lowest 
to hi^hest fv-ide. 


The output is how 
ovdeved -fvom hijjhest 
to lowest pvide. 


Python 3.5.2 Shell 


»> 

»> product = {'Book': 4 9.99, 

'PDF' : 43.99, 

'Video ' : 199.99} 

»> 

»> for k in product: 

print(k, product[k]) 


PDF -> 43.99 
Book -> 49.99 
Video -> 199.99 
»> 

»> for k in sorted (product) : 

print(k, product[k]) 


"B" to*es beW "P", which tones be&re V" 
garh ^ h k ^~tbe detault—works. 


^etihe the dictionary 
^e^r °rder 

,s **o{* n,aintained) 


Addm^ a call to "sorted'' sorb 
the dictionary by key, which 
»ay or may not be what yow 
want here- 


Book -> 49.99 
PDF -> 43.99 
Video -> 199.99 
»> 

»> for k in sorted (product, key=product.get): 
print(k, product[k]) 

The additio^ o-P the 

PDF -> 43 99 ‘V lets 

Book -> 49 99 ^ by value- 

Video -> 199.99 
»> 

»> for k in sorted (product, key=product.get, reverse=True) : 
print(k, product[k]) 


Video -> 199.99 
Book -> 49.99 
PDF -> 43.99 

»> I 


Addir >5 the 'Vevevse" 
a\r^umeir\t *£li^S the 
sovt ovdcv. 


Ln: 284 Coi: 4 



Learn more about how to sort with Python from this wonderful HOWTO : 

https://docs.python, org/3/hoivto/sorting. html#sortinghomto 
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6. More frow the Standard Ubrary 

Python’s Standard library is full of goodness. It’s always a worthy exercise to take 20 
minutes every once in a while to review what 5 s available, starting here: 


https://docs.python. org/3/library /index, html 


If what you need is in the Standard library , don’t waste your precious time rewriting 
it. Use (and/or extend) what 5 s already available. In addition to the Python docs, Doug 
Hellmann has ported his popular Module of the Week material over to Python 3. Find 
Doug’s excellent material here: 

https://pymotiv. com/3/ 


WeVe reviewed a few of our favorite Standard library modules below. Note that we can’t 
stress enough how important it is to know what’s in the Standard library , as well as what 
all the provided modules can do for you. 


collectioris 


This module provides importable data structures, over and above the built-in list, tuple 
dictionary, and set. There’s lots to like in this module. Here 5 s an abbreviated list of 
what’s in collections: 

• OrderedDict:A dictionary that maintains insertion order. 

• Counter: A class that makes counting things almost too easy. 

• ChainMap: Combines one or more dictionaries and makes them appear 
as one. 



itertools 

You already know Python’s for loop is great, and when reworked as a comprehension, 
looping is crazy cool. This module, itertools, provides a large collection of tools 
for building custom iterations. This module has a lot to offer, but be sure to also check 
out product, permutations, and combinations (and once you do, sit back 
and thank your lucky stars you didn’t have to write any of that code). 



functools 

The functools library provides a collection of higher-order functions (functions 
that take function objects as arguments). Our favorite ispartial, which lets you 
“freeze” argument values to an existing function, then invoke the function with a new 
name of your choosing. You won’t know what you’re missing until you try it. 
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doing many things 


1 Ruwwmg Your Code Concurrently 


In Ghapter 11 3 A, you used a thread to solve a waiting problem. Threads are not the 
only game in town when it comes to running code concurrently within your programs, 
although, to be honest, threads are the most used and abused of all of the available 
techniques. In this book, we deliberately kept our use of threads as simple as possible. 


There are other technologies available to you when you find yourself in a situation 
where your code has to do more than one thing at once. Not every program needs 
these types of Services, but it is nice to know that Python has a bunch of choices in this 
area should the need arise. 

In addition to the threading module, here are some modules worth checking out 
(and we also refer you back one page to #6, as Doug Hellmann has some great posts on 
some of these modules): 


You do realize theres only 
one of me, right? Yet you 
expect me to perform and 
understand multiple computing 
tasks at once?!?!? 


multiprocessing: This module allows you to spawn multiple Python 
processes, which—if you have more than one GPU core—can spread your 
computational load across many GPUs. 


• asyncio: Lets you specify concurrency via the creation and specification 
of coroutines. This is a relatively new addition to Python 3, so—for many 
programmers—it 5 s a very new idea (and the jury is stili out). 

• concurrent, futures: Lets you manage and run a collection of tasks 
concurrently 

Which of these is right for you is a question youil be able to answer once you Ve tried 
each of them with some of your code. 



New keywords: async and await 

The async and await keywords were added in Python 3.5, and provide a Standard 
way to create coroutines. 

The async keyword can be used in front of the existing for, with, and def 
keywords (with the def usage receiving the most attention to date). The await 
keyword can be used in front of (almost) any other code. As of the end of 2016, 
async and await are very new, and Python programmers the world over are only 
just beginning to explore what they can do with them. 

The Python docs have been updated with information on these new keywords, but, 
for our money, youTl find the best descriptions of their use (and the craziness that 
using them induces) by searching TouTube for anything on the topic by DavidBeazley. 
Be warned: David’s talks are always excellent, but do tend to lean toward the more 
advanced topics in the Python language ecosystem. 

David’s talks on Python 5 s GIL are regarded as classics by many, and his books are great 
too; more on this in Appendix E. 



"GIL"stands for "Global 
Interpreter Lock". The GIL is 
an internal mechanism used 
by the interpreter to ensure 
stability. Its continued use 
within the interpreter is the 
subject of much discussion 
and debate within the 
Python community. 
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8. ftUls with Tkiwter (and Fuw with Turtlcs) 

Python comes with a complete library called tkinter (the Tk interjace ) for building 
cross-platform GUIs. You may not realize it, but youVe been using an application from 
the very first chapter of this book that is built with tkinter: IDLE. 

What 5 s neat about tkinter is that it comes preinstalled (and ready for use) with every 
Python installation that includes IDLE (i.e., nearly all of them). Despite this, tkinter 
doesnl receive the use (and love) it deserves, as many believe it to be unnecessarily 
clunky (compared to some third-party alter nati ve s). Nevertheless, and as IDLE 
demonstrates, it is possible to produce useful and usable programs with tkinter. 

(Did we mention that tkinter comes preinstalled and ready for use?) 

One such usage is the turtle module (which is also part of the Standard library). To 
quote the Python docs: Turtle graphics is a popular way for introducing programming to 
kids. It was part of the original Logo programming language developed by Wally Feurzig and 
Seymour Papert in 1966. Programmers (i.e., mainly kids, but fun for newbies, too) can 
use commands like lef t, right, pendown, penup, and so on to draw on a GUI 
canvas (provided by tkinter). 

Here’s a small program, which has been adapted ever so slightly from the example that 
comes with the turtle docs: 


f\s> weII as show’m$ w *turtlc m 
adiion, IW.s small probam 
also demonstvates Ibe use 
ot Pythoft's "vjW ile loop a»d 
"bv-eak" staiement Tbey u/ov-k - 
e *ad-tly as Wd e*pedt tbe» 

{o, but do» t see »eavl7 as mudb 
ad-tio» as ibe w W Ioof a»d 

eom\>v*eKeir\s'«oy\s. 


# # # tksample.py - /Users/paul/Documents/tksample.py (3.5.2) 

from turtle import * 


color(' purple' , 'cyan' ) 

begin_fill() 


while True: 

forward(200) 
left(170) 

if abs (pos()) < 1: 
break 


|end_f ill () 
done () 


Ln: 13 Coi: 0 


And when this small turtle program is executed, a thing of beauty is drawn and 
appears on screen: 



We khow you ea^ do bettev* 
*tKah so why ho*b Aive 
U iu\rtle w a tvy? 
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test test test 


9. It'$ Not Over Til lt'$ Tested 


This book has barely mentioned automated testing, aside from a passing nod to 
the py .test tool for checking conformance to PEP 8 (at the end of Chapter 
4). This is not because we think automated testing isn’t important. We think 

automated testing is very important. It is such an important topic that 
entire books are dedicated to it. 


That said, in this book, we avoided automated testing tools on purpose. This 
has nothing to do with how we feel about automated testing (it really is 
very important). However, when you are first learning to program in a new 
programming language, introducing automated testing can confuse more than 
it clarifies, as the creation of tests assumes a good understanding of the thing 
being tested, and if that “thing” happens to be a new programming language that 
you’re learning...well, you can see where we’re going with this, can’t you? Its a bit 
like the chicken and the egg. Which comes first: learning to code, or learning how 
to test the code you’re learning? 

Of course, now that you’re a bona-fide Python programmer, you can take the 
time to understand how Python’s Standard library makes it easy to test your code. 
There are two modules to look at (and consider): 



• doctest: This module lets you embed your tests in your module’s 
docstrings, which isn’t as weird as it sounds and is very useful. 

• unittest: You may have already used a “unittest” library with 
another programming language, and Python comes with its very own 
version (which works exactly as you’d expect it to). 

The doctest module is adored by those who use it. The unittest module 
works like most other “unittest” libraries in other languages, and a lot of Python 
programmers complain that its not pythonic enough. This has led to the creation 
of the hugely popular py. test (which we talk more about in the next appendix). 



Don't you dare f lip on over to the 
next page before youve thought about 
how youd use Pythohs automated 
testing tools to check the correctness 
of the code you write. 
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10. Pebug, Pebug, Pebug 


You’d be forgiven for thinking that the vast majority of Python programmers 
revert to adding print calls to their code when something goes wrong. And you 
wouldn’t be far off: it’s a popular debugging technique. 

Another is experimenting at the >>> prompt, which—if you think about it—is 
very like a debugging session without the usual debugging chores of watching 
traces and setting up breakpoints. It is impossible to quantify how productive the 
>>> prompt makes Python programmers. All we know is this: if a future release 
of Python decides to remove the interactive prompt, things will get ugly. 

If you have code that’s not doing what you think it should, and the addition of 
print calls as well as experimenting at the >>> prompt have left you none the 
wiser, consider using Python 5 s included debugger: pdb. 

It’s possible to run the pdb debugger directly from your operating systenis 
terminal window, using a command like this (where myprog . py is the program 
you need to fix): 



python3 -m pdb myprog.py 


It 5 s also possible to interact with pdb from the >>> prompt, which is as close an 
instantiation of “the best of both worlds” as we think youil ever come across. 
The details of how this works, as well as a discussion of all the usual debugger 
commands (set a breakpoint, skip, run, etc.) are in the docs: 


You can learn all 
about traces and 
breakpoints by 
workingf through 
the " pdb” docs. 


A s diways, W\t\do ws 
uscv-s hccd "to use 

'Vy 'mstedd of 

"python" CThaCs tt P y", 

space, -then ndhus Z). 


https://docs.python.Org/3/library/pdb.html 


The pdb technology is not an “also ran,” nor was it an afterthought; it 5 s a 
wonderfully feature-full debugger for Python (and it comes built-in). 


Make suV " c 

a 

uvtdicvstandm^ 

Py-tho^s w fdb - 
dcbu^cv is pav-t 
o-f youv -toolkit 
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X Even More Tools, + 
Libraries, and Modules 



We know what you’re thinking as you read this appendix’s title. 

Why on Earth didn’t they make the title of the last appendix: The Top Twenty Things We 
Didrit Couer? Why another 10? In the last appendix, we limited our discussion to stuff that 
comes baked in to Python (part of the language’s “batteries included”). In this appendix, we 
cast the net much further afield, discussing a whole host of technologies that are available 
to you because Python exists. There’s lots of good stuff here and—just like with the last 
appendix—a quick perusal won’t hurt you one single bit. 


this is an appendix 
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ipython ipython ipython 


1. Alternatives to »> 

Throughout this book weVe happily worked at Python’s built-in >>> prompt, 
either from within a terminal window or from within IDLE. In doing so, we hope 
weVe demonstrated just how effective using the >>> prompt can be when you’re 
experimenting with ideas, exploring libraries, and trying out code. 

There are lots of alternatives to the built-in >>> prompt, but the one that gets the 
most attention is called ipython, and if you find yourself wishing you could do more 
at the >>> prompt, ipython is worth a look. It is very popular with many Python 
programmers, but is especially popular within the scientific community. 

To give you an idea of what ipython can do compared to the plain of >>> prompt, 
consider this short interactive ipython session: 


^ paul — IPython: Users/paul — ipython — 79x19 


VVi-bV» "ipythoin” 
ms-talled, stav-t «t 
at yoi*v operati^ 
s yste*n S dommaind l» r 'C’ 


login: Tue Aug 3^20:23:36 on ttysOOl 
[MacBook-Pro-6:~ paul$ Ipython 

Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 26 2016, 10:47:25) 

Type "Copyright”, "credits" or "license" for more information. 

IPython 5.1.0 — An enhanced Interactive Python. 

? -> Introduction and overview of IPython 1 s features. 

%quickref -> Quick reference. 

help -> Python's own help system. 

object? -> Details about 'object', use 'object??' for extra details 


[In [1]: nums = [1, 2, 3] 





[In [2]: squares = [n*n for n in nums] 


[In [3]: squares 
Out[ 3 ]: [1, 4, 9] 


In [4]: 


You ea* easily teli >n\\\c\\ output 
^oes with whieh iir^ut (thav>ks 
*u**loev-ed pvompts). 



Vou\r Lo&t is eolov-eoded- 


Find out more about ipython at https://ipython.org. 

There are other >>> alternatives, but the only other one thaf s a match (in our view) 
for what ipython has to offer is ptpython (more information can be found here: 
https://pypi.org/project/ptpython/). If you like working within a text-based terminal 
window, but are looking for something a bit more “full screen” than ipython, take a 
look at ptpython. You won’t be disappointed. 


As witk all tkird- 
party modules, you 

« • w . 

can use nip to 
install kotk "ipytko 
and "ptpytkon”. 
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2 . Altcrwatives to IPIE 


We’re not afraid to state this: we have a soft spot for IDLE. We really like the fact 
that Python not only comes with a capable >>> prompt, but also ships with a 
passable cross-platform GUI-based editor and debugger. There are few other 
mainstream programming languages that provide anything similar as part of 
their default install. 


Regrettably, IDLE gets a fair amount of flack in the Python community, as it 
stacks up poorly against some of the more capable “professional” offerings. We 
think this is an unfair comparison, as IDLE was never designed to compete in that 
space. IDLE' s main goal is to get new users up and going as quickly as possible, 
and it does this in spades. Consequently, we feel IDLE should be celebrated more 
in the Python community 


IDLE aside, if you need a more professional IDE, you have choices. The most 
popular in the Python space include: 

• Eclipse: https://www. eclipse. org 

• PyCharm : https://www.jethrains. com/pycharm/ 

• WingWare : https://wingware.com 

Eclipse is a completely open source technology, so won’t cost you more than the 
download. If you’re already an Eclipse fan, its support for Python is very good. 
But, if you aren’t currently using Eclipse , we wouldn’t recommend its use to you 
due to the existence of Py Charm and WingWare. 

Both Py Charm and WingWare are commercial products, with “community 


.55 


versions available for download at no cost fbut with some restrictionsf Unlike 


Eclipse , which targets many programming languages, both Py Charm and 
WingWare target Python programmers specifically and, like all IDEs, have great 
support for project work, links to source code management tools (likegzi), support 
for teams, links to the Python does, and so on. We encourage you to try both, 
then make your choice. 

If IDEs aren’t for you, fear not: all of the workfs major text editors offer 
excellent language support to Python programmers. 


What does Paul use? 

Paufs text editor of choice is vim (Paul uses MacVim on his development 
machines). When working on Python projects, Paul supplements his use of vim 
with ptpython (when experimenting with code snippets), and he 5 s also a fan of /V)*-. PyChaem 
IDLE. Paul uses git for local version control. 



/W- iVm^WeJv- 


For what it’s worth, Paul doesn’t use a full-featured IDE, but his students love 
PyCharm. Paul also uses (and recommends) Jupyter Notebook, which is discussed 
next. 
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jupyter jupyter jupyter 


$. Jupyter Notebook: The Web-Pased IPE 


In item #1, we drew your attention to ipython (which is an excellent >>> 
alternative). From the same project team comes Jupyter Notebook (previously known 
as iPython Notebook). N 


Jupyter Notebook can be described as the power of ipython in an interactive 
web page (which goes by the generic name of “notebook”). What’s amazing 
about Jupyter Notebook is that your code is editable and runnable from within the 
notebook, and—if you feel the need—you can add text and graphies, too. 


Here’s some code from Ghapter 12 running within a Jupyter Notebook. Note how 
weVe added textual descriptions to the notebook to indicate what 5 s going on: 


"The ne%t gelatio* ot 
yPykr Uotebook h talled 
Ju rytey Lab, and it was 
' n a lfba" as work on -fchis 
book was dondluding. kfeep 
an eye out £*• the Jupytev 
i-ab pv-oject' it's going to be 
somethmg vathe^- spedial. 


® ® ® 0 Home x^ £ Appendix D x \\^ \ 


• 

C O localhost:8888/notebooks/Appendix%20D.ipynb#Playing-with-the-url_utils.py 


☆ : 

^2 J U pytG r Appendix D (autosaved) 

& 


File Edit View Insert Cell Kernel Help 

£ | Python 3 O 


E + «K f?] C) ^ ^ HIC Code i mD CellToolbar 




Playing with the code from url_utils.py 

Define the <b>" gen_from_urls' '</b> generator function: 

In [ 1 ] : import requests 

def gen_from_urls (uris: tuple) -> tuple: 

for resp in (requests.get(uri) for uri in uris): 

yield len (resp.content), resp.statuscode, resp.uri 


Let's specify three URLs to work with: 

In [2]: uris = (' http://jupyter.org/' , 'http://ipython.org/', 'http://pydata.org/') 
Use the generator function with a for loop: 


In [3]: for _, status, _ in gen_from_urls(uris): 
print (status) 


200 

200 

200 



Learn more about Jupyter Notebook from its website ( http://jupyter.org ), and use pip 
to install it onto your computer, then start exploring. You will be glad you did. 
Jupyter Notebook is a killer Python application. 
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4. Poing Pata Science 


When it comes to Python adoption and usage, there’s one domain that 
continues to experience explosive growth: the wodd of data Science. 


This is not an accident. The tools available to data scientists using Python are 
world class (and the envy of many other programming communities). What 5 s 
great for non-data scientists is that the tools favored by the data folks have 
wide applicability outside the Big Data landscape. 

Entire books have been (and continue to be) written about using Python 
within the data Science space. Although you may think this advice biased, the 
books on this subject from 0’Reilly Media are excellent (and plentiful). 0’Reilly 
Media has made a business out of spotting where the technology industry is 
heading, then ensuring there’s plenty of great, high-quality learning material 
available to those wanting to learn more. 


& 0 m /Vfavke-tihg.- 

This means -fchey go-fc 
°u\r r»er*o. © 



Here’s just a selection of some of the libraries and modules available to you if 
you do data Science (or any other Science calculations, for that matter). If data 
Science isn’t your thing, check out this stuff anyway—there 5 s lots to like here: 

• bokeh: A set of technologies for publishing interactive graphics 
on web pages. 

• matplotlib/seaborn: A comprehensive set of graphing 
modules (which integrates with ipython andJupyterNotebook). 

• numpy: Among other things, allows you to efficiently sto re and 
manipulate multidimensional data. If you 5 re a fan of matrices, 
you ? 11 love numpy. 

• scipy: A set of scientific modules optimized for numerical data 
analysis, which complements and expands upon what’s provided 
by numpy. 

• pandas: If you are coming to Python from the R language, then 
youdl feel right at horne with pandas, which provides optimized 
analysis data structures and tools (and is built on top of numpy 
and matplotlib). The need to use pandas is what brings a 
lot of data folk to the community (and long may this continue), 
pandas is another killer Python application. 

• scikit-learn:Asetof machine learning algori thms and 
technologies implemented in Python. 

Note: most of these libraries and modules are pip-installable. 

The best place to start learning about the intersection of Python and data 
Science is the PyData website: http://pydata.org. Glick on Downloads , then marvel 
at what 5 s available (all as open source). Have fun! 
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5 . Web Pevelopment Technologies 


Python is very strong in the web space, but Flask (with Jinja2) isn’t the only game 
in town when it comes to building server-side webapps (even though Flask is a 
very popular choice, especially if your needs are modest). 

The best-known technology for building webapps with Python is Django. It 
wasn’t used in this book due to the fact that (unlike Flask) you have to learn 
and understand quite a bit before you create your first Django webapp (so, for a 
book like this, which concentrates on teaching the basies of Python well , Django 
is a poor fit). That said, there’s a reason Django is so popular among Python 
programmers: it 5 s really, really good. 

If you class yourself as a “web developer,” you should take the time to (at the very 
least) work through Django ’s tutorial. In doing so, you’11 be better informed as to 
whether you’11 stick with Flask or move to Django. 


If you do move to Django , you’11 be in very good company: Django is such a large 
community within the wider Python community that it’s able to sustain its own 
conference: DjangoCon. To date, DjangoCon has occurred in the US, Europe, and 
Australia. Here are some links to learn more: 


Djanjo 5 s landing page (which has a link to the tutorial 

https://www, djangoproject. com 

DjangoCon US: 

https://djangocon, us 

DjangoCon Europe: 

https: //djangocon, eu 

DjangoCon Australia: 

http://djangocon, com, au 


Put wait, there'$ more 

As well as Flask and Django , there are other web frameworks (and we know we’11 
neglect to mention somebody 5 s favori te). Those we hear the most about include: 
Pyramid , Turbo Gears, web2py , CherryPy , and Bottle. Find a more complete list on the 
Python wiki: 

https://wiki,python, org/moin/WebFrameworks 
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6. Working with Web Pata 


In Chapter 12, we briefly used the requests library to demonstrate just how cool 
our generator was (compared to its equivalent comprehension). Our decision to use 
r eques ts was no accident. If you ask most Python developers working with the Web 
what their favorite PyPI module is, the majority responds with one word: “requests.” 

The requests module lets you work with HTTP and web Services via a simple, yet 
powerful, Python API. Even if your day job doesn’t involve working directly with 
the Web, youdl learn a lot just from looking at the code for requests (the entire 
requests project is regarded as a master class in how to do things the Python way). 

Find out more about requests here: 


PyPI: Tke Pytkon 
Package Inaex lives 
at https://pypi.org/. 


http://docs.python-requests. org/en/master/ 


Scrape that web data! 

As the Web is primarily a text-based platform, Python has always worked well in that 
space, and the Standard library has modules for working with JSON, HTML, XML, 
and the other similar text-based formats, as well as all the relevant Internet protocols. 
See the following sections of the Python docs for a list of modules that come with the 
Standard library and are of most interest to web/Internet programmers: 


Internet Data Handling: 

https://docs,python, org/3/library/ netdata.html 

Structured Markup Processing Tools: 

https://docs,python, org/3/library/markup, html 

Internet Protocols and Support: 

https://docs.python, org/3/library/internet, html 


If you find yourself having to work with data that 5 s only available to you via a static 
web page, youdl likely want to scrape that data (for a quick scraping primer, see https:// 
en.wikipedia.org/wiki/Web_scraping ) . Python has two third-party modules that will save 
you lots of time: 


Beautiful Soup: 

https://ivwzv. crummy. com/softivare /BeautifulSoup / 

Scrapy: 

http://scrapy. org 


Soup? Soup! Did 
somebody mention Soup? 
And they said it was 
"beautiful"... mon dieu. 


O 




Try both, see which one solves your problem best, and then get on with whatever else 
needs doing. 


more than mysql 


7 More Pata Sources 

To keep things as real as possible (while trying to keep it simple), we used MySQL 
as our database backend in this book. If you spend a lot of time working with 
SQL (regardless of the database vendor you favor), then stop whatever you’re 
doing and take two minutes to use pip to install sqlalchemy—it may be your 
best two-minute installation ever. 

The sqlalchemy module is to SQL geeks what requests is to web geeks: 
indispensable. The SQL Alchemy project provides a high-level, Python-inspired 
set of technologies for working with tabular data (as stored in the likes of MySQL , 
PostgreSQL , Oracle , SQL Server , and so on). If you liked what we did with the DBcm 
module, you’re going to love SQL Alchemy , which bilis itself as the database toolkit 
for Python. 

Find out more about the project at: 

http://www. sqlalchemy. org 


There'$ wore to queryiwg data thaw SCII 

Not all the data youll ever need is in an SQL database, so there will be times 
when SQL Alchemy won’t do. NoSQL database backends are now accepted as a 
valid addition to any data center, with MongoDB serving as the classic example as 
well as the most popular choice (even though there are many). 

If you end up working with data that 5 s being presented to you as JSON, or in a 
nontabular (yet structured) format, MongoDB (or something similar) may be just 
what you’re looking for. Find out more about MongoDB here: 


https://www. mongodh. com 


And check out the Python support for programming MongoDB using the 
pymongo database driver from the PyMongo documentation page: 


https://api. mongodh. com/python/current/ 


No matter where 
our data is—in an SQL 
or NoSQL data store— 
Python and its third-party 
modules do the trick. 
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8. Programming tools 

No matter how good you think your code is, bugs happen. 

When they do, Python has lots to help you: the >>> prompt, the pdb 
debugger, IDLE , print statements, unittest, and docte st. When these 
options aren’t enough, there are some third-party modules that might help. 

Sometimes, you’11 make a classic mistake that everyone else has made before 
you. Or perhaps you Ve forgotten to import some required module, and the 
problem doesn’t crop up until you’re showing off how great your code is to a 
room full of strangers (whoops). 

To help avoid this type of thing, get PyLint , Python 5 s code analysis tool: 


https://www.pylint. org 


PyLint takes your code and telis you what might be wrong with it before you 
run it for the first time. 

If you use PyLint on your code before you run it in front of a room full of 
strangers, it may very well prevent blushing. PyLint might also hurt your 
feelings, as no one likes to be told their code is not up to scratch. But the 
pain is worth the gain (or maybe that should be: thepain is better than thepublic 
emb arras sment). 

More help wlth testing, too 

In Appendix C, #9, we discussed the built-in support Python provides for 
automated testing. There are other such tools, too, and you already know that 
py. te st is one of them (as we used it earlier in this book to check our code 
for PEP 8 compliance). 

Testing frameworks are like web frameworks: everyone has their favorite. That 
said, more Python programmers than not favor py. te st, so we’d encourage 
you to take a closer look: 


http://doc.pytest . org/en/latest/ 


I think someones a little 
conf used. I was told to "clean 
up the lint"...but I think they 
meant for me to use PyLint 
to clean up your code. What a 
mess.Jf only youd written a 
py.test first, eh? 
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kivy kivy kivy 


9. Kivy: Our Pick for "Coolest Project Ever" 

One area where Python is not as strong as it could be is in the world of mobile touch 
devices. There are a lot of reasons why this is (which we arefft going to get into here). 
Suffice it to say, at the time of publication, it is stili a challenge to create an Android or 
iOS app with Python alone. 

One project is attempting to make progress in this area: Kivy. 

Kivy is a Python library that allows for the development of applications that use 
multitouch interfaces. Pop on over to the Kivy landing page to see what 5 s on offer: 


https://kivy. org 


Once there, click on the Gallery link and sit back for a moment while the page loads. If 
a project grabs your eye, click on the graphic for more information and a demo. While 
you view the demo, keep the following in mind: everythingyou are looking at was coded with 
Python. The Blog link has some excellent material, too. 

What 5 s really cool is that your Kivy user interface code is written once, then deployed 
on any supported platform unchanged. 

If you are looking around for a Python project to contribute to, consider donating 
your time to Kivy : it’s a great project, has a great team working on it, and is technically 
challenging. If nothing else, you wofft be bored. 


/\ s»apsh°t s 

UdmO) ya^e &*'«' 

2.0 l& oy\C oAr 

i\\t \v deyloy^CK-ts: a 

-fully ir***ev-sWe -touth 




• © Kivy: Cross-platform Python x 

<- C A https://kivy.0rg/#h0me 


wy kivy 


O a g + f • 

Home Download Gallery Help Organization About Blog 


Plane White 

Interactive white surface for discovering art exhibition. 
View more projects in the Gallery » 


Kivy - Open source Python library for rapid development of applications 
that make use of innovative user interfaces, such as multi-touch apps. 
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10. Alternative Implementations 

You already know from item #1 in Appendix C that there’s more than one Python 
language release (Python 2 and Python 3). This means that there’s at least two Python 
interpreters: one that runs Python 2 code, and one that runs Python 3 code (which 
is the one weVe used throughout this book). When you download and install one 
of the Python interpreters from the Python website (like you did in Appendix A) , the 
interpreter is referred to as the CPython reference implementation. CPython is the version 
of Python distributed by the Python cor e developers , and takes its name from the fact that 
it’s written in portable G code: it’s designed to be easily ported to other computing 
platforms. As you saw in Appendix A, you can download installers for Windows and Mac 
OS X, as well as find the interpreter preinstalled within your favorite Linux distribution. 
All of these interpreters are based on CPython. 

Python is open source, so anyone is free to take CPython and change it in any way 
they wish. Developers can also take the Python language and implement their own 
interpreter for it in whichever programming language they wish, using whichever 
compiler techniques they like, running on whatever platform theyVe using. Although 
doing all of this is not for the faint of heart, plenty of developers do this (some of 
them describe it as “fun”). Here are short descriptions and links to some of the more 
active projects: 






PyPy (pronounced “pie-pie”) is a experimental compiler testbed for Python 2 
(with Python 3 support on the way). PyPy takes your Python code and runs 
it through a just-in-time compilation process, producing a final product that 
runs faster than CPython in many instances. Find out more here: 

http://pypy . org 


IronPython is a version of Python 2 for the .NET platform: 

http:// ironpython.net 


Jython is a version of Python 2 that runs onJavaSJVM: 

http://zvzvzv.jython. org 


MicroPython is a port of Python 3 for use on the pyboard microcontroller, which 
is no bigger than your two thumbs side by side, and may well be the coolest 
little thing you Ve ever seen. Take a look: 



http://micropy thon. org 


Despite all these alternative Python interpreters, the majority of Python programmers 
remain happy with CPython. Increasingly, more developers are choosing Python 3. 
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X The Python Community 



Python is much more than a great programming language. 

It’s a great community, too. The Python Community is welcoming, diverse, open, friendly, 
sharing, and giving. We’re just amazed that no one, to date, has thought to put that 
on a greeting card! Seriously, though, there’s more to programming in Python than the 
language. An entire ecosystem has grown up around Python, in the form of excellent 
books, blogs, websites, conferences, meetups, usergroups, and personalities. In this 
appendix, we take a survey of the Python community and see what it has to offer. Don’t 
just sit around programming on your own: get involved! 
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gotta love guido 


PPFl: Pewevolewt Pictator for Life 

Guido van Rossum is a Dutch programmer whose gift to the world is the Python 
programming language (which he started as a “hobby” in the last 1980s). The ongoing 
development and direction of the language is set by the Python core developers , of which 
Guido is but one (albeit a very important one). Guido’s title of Benevolent Dictator for Life is in 
recognition of the Central role he continues to play in the day-to-day life of Python. If you 
see the letters BDFL in relation to Python, that’s a reference to Guido. 

Guido is on the record as stating that the name “Python” is a nod (and a wink) toward the 
British television comedy troupe Monty Pythoni Flying Circus , which helps explain the use of 
the name spam for many of the variables referred to in the Python docs. 

Despite Guido’s leading role, he does not own Python: nobody does. However, the interests 
of the language are protected by the PSF. 


PSF: The Python Software Foundation 

The PSF is a nonprofit organization that looks after the interests of Python, and is run by 
a nominated/elected board of directors. The PSF promotes and sponsors the continued 
development of the language. This is from the PSF’s mission statement: 

The mission of the Python Software Foundation is to promote, protect, and advance the Python 
programming language, and to support andfacilitate the growth of a diverse and international 
community of Python programmers. 

Anyone can join the PSF and get involved. See the PSF website for details: 

https://www.python. org/psf/ 

One of the PSF’s major activities is involvement in (and the underwriting of) the annual 
Python conference: PyCon. 


Have your say: 

join the PSF. 


PyCon: The Python Conference 

Anyone can attend (and speak at) PyCon. In 2016, Portland, Oregon, hosted the 
conference, with thousands of Python developers in attendance (the previous two PyCons 
were held in Montreal, Ganada). PyCon is the largest Python conference, but not the only 
one. You’11 find Python conferences across the globe, ranging in size from small, regional 
conferences (tens of attendees), through national conferences (hundreds of attendees), up 
to the likes of EuroPython (thousands of attendees). 

To see if there’s a PyCon near you, search for the word “PyCon” together with the name 
of your nearest city (or the country you live in). Ghances are, you’11 be pleasantly surprised 
by what you find. Attending a local PyCon is a great way to meet and interact with like- 
minded developers. Many of the talks and sessions at the various PyCons are recorded: pop 
over to TouTube and type “PyCon” for an idea of what’s available to view. 


Get involveT 
attend PyCon. 
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A Tolerant Community: Respect for Piversity 

Of all the programming conferences that exist today, PyGon was one of the first to 
introduce and insist on a Code of Conduct. You can read the 2016 Gode of Gonduct 
here: 


https: //us.pycon. org/ 2016 /about/code-of-conduct/ 


Such a development is a very good thing. More and more, the smaller regional PyCons 
are adopting the Gode of Gonduct, too, which is also very welcome. A community 
grows to be strong and inclusive when there are ciear guidelines about what’s 
acceptable and what isn’t, and the Gode of Gonduct helps to make sure all the world’s 
PyCons are as welcoming as they can be. 

In addition to striving to ensure everyone is welcome, a number of initiatives attempt 
to increase the representation of specific groups within the Python community, 
especially where—traditionally—such groups have been underrepresented. The 
best-known of these is PyLadies , which was established per their mission to help “more 
women become active participants and leaders in the Python open source community” 
If you’re lucky, there 5 s a PyLadies “chapter” near you: find out by starting your search 
from the PyLadies website: 


http://www.pyladies. com 


Just like the Python community, PyLadies started out small, but has very quickly grown 
to have global reach (which is truly inspirational). 


Come for the language, stay for the community 

Many programmers new to Python comment on how inclusive the Python community 
is. A lot of this attitude stems from Guido 5 s guiding hand and example: firm, yet 
benevolent. There are other leading lights, too, and plenty of inspirational stories. 

It doesn’t get much more inspirational than Naomi CedePs talk at EuroPython (which 
was repeated at other regional conferences, including PyCon Ireland). Here 5 s a link to 
Naomfs talk, which we encourage you to watch: 


Encourage 
and support 
diversity 
witnin the 
Python 

community. 


https://wwwyoutube . com/watch? v -cCCiA-IlVco 


Naomfs talk surveys a life in Python, and discusses how the community supports 
diversity, and how there 5 s always more work for everyone to do. 

One way to learn more about a community is to listen to some of the podcasts 
generated by its participants. We discuss two Python podcasts next. 
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Python Podcasts 


There are podcasts on everything these days. Within the Python community there 
are two we feel are well worth subscribing and listening to. Whether it’s something 
to listen to while driving, cycling, running, or chilling out, these podcasts are both 
deserving of your attention: 

• Talk Python to Me: https://talkpython.fm 

• Podcast. init : http://pythonpodcast.com 

Follow both of these podcasts on Twitter , teli your friends about them, and give 
the producers of these podcasts your full support. Both Talk Python To Me and 

Podcast. _ init _are produced by regular members of the Python community for 

the benefit of all of us (and not for profit). 


Python Newsletters 

If podcasts aren’t your thing, but you stili want to keep up with what’s happening 
in the Python world, there are three weekly newsletters that can help: 

• Pycoder’s Weekly: http://pycoders.com 

• Python Weekly: http://www.pythonweekly.com 

• Import Python: http://importpython.com/newsletter 

These curated newsletters provide links to all types of material: blogs, vlogs, 
articles, books, videos, talks, new modules, and projects. And their weekly 
announcements arrive right to your email inbox. So, go ahead and sign up. 



As well as a foundation, multiple conferences, subgroups like PyLadies , codes of 
conduct, recognition of diversity, podcasts, and newsletters, Python also has its 
very own notion of %en. 
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The Zew of Python 

Many moons ago, Tim Peters (one of Pythorfs early leading lights) sat 
down and wondered: what is it that makes Python Python? 

The answer came to Tim as The fen of Python , which you can read 
by starting any version of the interpreter and typing the following 
incantation into the >>> prompt: 

import this 

WeVe done this for you, and shown the output of the above line of code 
in the screenshot at the bottom of this page. Be sure to read The fen of 
Python at least once a month. 

Many have tried to compress The fen of Python into something a little 
easier to digest. None other than xkcd has given it a go. If you’re 
connected to the Internet, type this line of code into your >>> prompt to 
see (quite literally) how xkcd got on: 

import antigravity 



Python 3.5.2 Shell 


»> 

»> import this 

The Zen of Python, by Tim Peters 

Beautiful is better than ugly. 

Explicit is better than implicit. 

Simple is better than complex. 

Complex is better than complicated. 

Flat is better than nested. 

Sparse is better than dense. 

Readability counts. 

Special cases aren't special enough to break the rules. 

Although practicality beats purity. 

Errors should never pass silently. 

Unless explicitly silenced. 

In the face of ambiguity, refuse the temptation to guess. 

There should be one-- and preferably only one --obvious way to do it. 
Although that way may not be obvious at first unless you're Dutch. 

Now is better than never. 

Although never is often better than *right* now. 

If the implementation is hard to explain, it's a bad idea. 

If the implementation is easy to explain, it may be a good idea. 
Namespaces are one honking great idea -- let's do more of those! 

»> 


Ln: 28 Coi: 4 



7 ^™ 


Remcr*bev : vead *this least^ on£e a 
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to read next 


Which Pook Should I Read Next? 


o 


Is that it? You can'+ 
finish off without giving me 
some recommendation on 
what I should read next. 
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Our Favorite Python Pooks 


As Python has grown in popularity the number of books devoted to the 
language has blossomed. Of all the books out there, there are two we regard as 
indispensable. 



sring Python 3 


Dovid Beozley & Bi 


Lu ciano Rarnalho 
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Cookbook- 


If dee p dives ave move youv 
™ ,h 9' vead -this ex^elleh-t book. 
Theres a lot in here, but it's 
all good (and you'|| be a better 
rythoh progra«,«,er (oy the 
experiende). 


you are here ► 


569 














Symbols 

>>>. See Python Shell 
<> (angle brackets) 256-257 
= (assignment operator) 13, 55, 72-74 
\ (backslash) 77 
A (caret) 192 
: (colon). See colon (:) 

, (comma) 54, 123, 134 
+ (concatenation operator) 543 
{ } (curly braces). See curly braces { } 

-= (decrement operator) 106 
/ (forward slash) 207 
+= (increment operator) 106,318 

* (multiplication operator) 87 

* notation 390-391 
** notation 392-393 

0 (parentheses). See parentheses 0 
[j (square brackets). See square brackets [j 
@ Symbol 207 

* Symbol 147 

% syntax 214, 543 
| (vertical bar) 262 

A 

Alt-P key combination (Linux/Windows) 31, 118 
angle brackets 256-257 
annotations (function) 162-163 
append method 58-59, 72, 270 


app.runQ function 207, 211, 217 
apt-get utility 527 
*args keyword 390, 401 

arguments 

about 147, 154-155 

adding multiple 165 

any number and type of 394 

by-address argument passing 184, 186-187 

by-reference argument passing 184, 186-187 

by-value argument passing 184—185, 187 

dictionary of 392—393 

function decorators 223, 390-395, 401 

interpreter processing 148 

list of 390 

methods and 317, 319-320, 322 
positional versus keyword assignment 171 
specifying default values for 170 

arrays. See lists 

arrow Symbol 162—163 

assignment operator 13, 55, 72-74 

assignment statements 13-14 

associative arrays. See dictionaries 

asterisks 390-393 

asyncio module 546 

async keyword 546 

Attribute Error exception 483 

attributes (state) 
about 49 

classes and 311-312,322 
dictionary lookup retrieves 369 
displaying 30 

Flask’s session technology and 368 
initializing values 323—325 
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the index 


methods and 322 
objects and 313, 315, 322 

authentication 364 
automated testing 548 
automatic reloading webapps 227 
await keyword 546 

B 

backslash 77 

BDFL (Benevolent Dictator for Life) 564 
Beazley, David 546, 569 
behavior. See methods (behavior) 

BIF (built-in functions) 161 

binary mode 247 

blocks of code. See suites of code 

bokeh library 555 

bool built-in function 156—158 

boolean values 116 

by-address argument passing 184, 186-187 
by-reference argument passing 184, 186-187 
by-value argument passing 184—185 

c 

call-by-reference semantics 184, 186 
call-by-value semantics 184—185 
CamelCase 312 
caret 192 

case sensitivity and conventions 116, 312 

cd command 175 

Geder, Naomi 565 

ChainMap class 545 

classes 

about 311—312 
attributes and 311-312, 322 
creating 310 

dehning functionality of 313—314 


empty 312,444 

methods and 311-312,318 

naming 312 

obj ects and 312-313 

with statement and 305, 310, 337-338 

class keyword 312 

@classmethod decorator 542 

client error messages 222 

close method 245-246 

Code of Gonduct 565 

collections module 545 

colon (:) 

blocks of code and 16-17 
comprehensions and 506 
dictionaries and 98, 123, 506 
functions and 149,162 
lists and 76 

combinations function (itertools module) 545 
comma 54, 123, 134 

command-line, running Python from 175—177, 190 

comments 147 

comparison operators 13,15 

compilation 7 

comprehensions. See also specihc types of loops 
about 493, 504, 517 
Bahama Buzzers example 478—517 
converting patterns into 491 
dictionary 493-496, 499-502, 506 
examining 492 

list 493-496, 504, 506, 508-511 
reading GSV data as dictionaries 480-484 
reading GSV data as lists 479 
set 504-505 

spotting patterns 489-490 
transforming data 484—486 
tuples and 504, 507 

concatenation operator 543 
concurrency options 465 
concurrent.futures module 546 
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connect function 291 
constant lists (tuples) 51 
constructor methods 323 
contextlib module 337 

context management protocol 
about 305—306, 310, 338—339 
creating a context manager 337, 339 
creating a context manager class 340 
exception handling and 440—441 
function decorators and 407 
initializing context manager class 338—342 
performing set-up 338-340, 343-344 
performing tear-down 338—340, 345 
readdressing webapp code 348-358 
testing context manager 346—347 

control statements 16 
copy method 7 3 
Gounter class 545 
GPython 561 

Gross-site Scripting (XSS) 419, 422 
GSV data 

Bahamas Buzzers example 478—482 
reading as dictionaries 480-484 
reading as lists 479 

csv module 479 

Gtrl-G key combination 210, 220 
Gtrl-P key combination (Mac) 31, 118 

curly braces { } 

blocks of code and 16 
comprehensions and 506 
dictionaries and 104,137-139 
sets and 123 
template engines 214 

current working directory 9—10, 174 
cursor method 291-295, 463-464 

D 

database-enabling webapps 

creating code to work with database and tables 296 


creating database and tables 287-295 
exception handling and 418, 420, 422, 440 
installing MySQL-Connector/Python 286 
installing MySQL datebase driver 285 
installing MySQL server 283 
introducing Python’s DB API 284 
reusing database code 301-306 
sharing code. See context management protocol 
storing data 300 

Data Science 555 

data structures 

built-in 13, 50, 161 
complex 135-142, 266-267 
copying 73 

dictionaries. See dictionaries 
lists. See lists 
sets. See sets 
tuples. See tuples 

datetime module 8, 11, 486 
day attribute (date.today) 11 
DB-API , 281, 284, xvi 
debugging 224, 549 

decorators, function. See function decorators 

decrement operator 106 

default values for arguments 170—171 

def statement 
about 147, 149 
async keyword and 546 
default values for arguments 170 
positional versus keyword assignment 171 

delimiters 77, 262, 506 
describe log command 289, 293 

dictionaries 

about 52, 103 

of arguments 392-393 

checking for membership in 117—119 

dictionaries within 136—140 

dynamic 114 

easy to read 97 

empty 104, 136, 161 


you are here ► 573 



the index 


frequency counts in 102—106,131 

growing at run-time 101 

iterating over 107 

iterating over keys and values 108 

iterating over rows of data 110 

key/value pairs and 52, 96, 115—120 

of lists 485-486 

reading GSV data as 480-484 

specifying ordering on output 109—110 

spotting in code 98 

spotting pattern with 489 

square brackets and 99-101 

dictionary comprehensions 493-496, 499-502, 506 
difference method 125,127 
dir built-in function 30, 259-260, 324 
distribution packages 178-182 
Django framework 203, 556 

docstring 
about 147 
adding 151, 168 
adding information to 162 
updating 165 

doctest module 548 

documenting functions 162—163 

dot-notation syntax 29, 58, 316 

dunder name 206, 238-239, 324-325, 338-345 

duplicate objects, sets and 53, 59 

dynamic assignment of variables 13, 48-49 

dynamic dictionaries 114 

dynamic lists 50-51, 62 

E 

Eclipse IDE 553 
editwindow 3—8,57, 150—151 
elif statement 17 
else statement 16-17,117 
embedded dictionaries 136-140 
embedded suites of code 18 


empty classes 312, 444 
empty dictionaries 104, 136, 161 
empty lists 55,58, 161 
empty sets 160—161 
empty statements 312 
empty strings 157 
empty tuples 161 
Enter key 22—24 

_enter_method 338—340, 443 

environ attribute (os module) 10 

environment variables 10 

escape characters 77, 257 

escape function (flask module) 257-258, 270 

escape function (html module) 11 

Exception class 427 

exception handling. See also specific exceptions 
built-in exceptions 427 
catch-all exception handler 428, 431 
context manager and 440—441 
creating custom exceptions 444—447 
databases and 418, 420, 422, 440, 448-455 
functions and 421-422 
import mechanisms 29 
indentation errors 45 
misspelled variables 45 
output display and 255 
PEP 8 failure messages 191-192 
run-time and 115—121,423—424,474 
syntax errors 5, 57 

webapps and 255, 418-420, 422, 433, 437-439 
with statement and 443, 451-452 

executing code. See also run-time 

Alt-P key combination for 31, 118 
Gtrl-P key combination for 31, 118 
F5 key for 4, 6, 151 
interpreter processing in 8 
invoking functions 150 
pausing execution 20, 28 
running concurrently 546 
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running immediately 7, 22 
running multiple times 20 

_exit_method 338-340, 443, 452-453 

extend method 64 
extends directive (Jinja2) 214 

F 

F5 key 4, 6, 151 
False value 156—157 

FileNotFoundError exception 423—424, 431 
Flask class 205,217 

Flask framework 
about 203, 556 

accessing HTML form data 226 
associating function with multiple URLs 236 
creating webapp objects 206 
debugging mode 224 
installing 202 

Jinja2 template engine 214-215, 229-230, 274, 276 

Markup object 257 

rendering templates from 217 

request object 226-227, 259 

running webapps 204-205 

session mechanism 367-368 

testing webapps 220-221 

threading and 471 

flask module 

escape function 257-258, 270 
Flask class 205,217 
session dictionary 368-379 

for loop. See also comprehensions 
about 20, 24-27, 504 
lists and 86-88, 479 
slices and 87 

spotting patterns in 489-490 
format method 543 

formatting 

data 484-486 
strings 543 


form dictionary (Flask) 226 
<form> tag 222 
forward slash 207 

frequency counts 

about 102—103, 131 
incrementing 105 
initializing 105 
selecting data structure 104 
updating 105-106 

function decorators 
about 209, 385 
adding 217-218 
adjusting behaviors 207 
arguments and 223, 390-395, 401 
components in writing 385—394 
context managers and 407 
creating 395-410 

URLs and 207, 209, 211, 218, 223, 396, 408 

function objects 386-389, 395, 397-398 

functions. See also arguments (functions); See also specihc 
functions 

about 9, 147—148 
best practice for 153 
built-in 161 
creating 149, 166—169 
documenting 162—163 
editing 150—151 

embedding generators within 511—516 
exception handling and 421—422 
importing 9, 28—29 
invoking 150 

invoking passed functions 387 
methods and 316,322 
modules and 9, 173 
multiple URLs 236 
naming 149, 165, 312 
nested 388, 400 
passing to functions 386 
returning from functions 389 
returning results 156—159 
reusing code with 146, 173 
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sharing 173 

string quote characters 152 
troubleshooting 184, 187 
variables and 321-322 

functools module 402, 545 

G 

generators 508, 510 
getcwd function (os module) 9—10 
GET method (HTTP) 222-223 
global variables 366 

H 

hashes. See dictionaries 
Hellman, Doug 545—546 
help command 31, 41, 66 
Homebrew package manager 283, 525 

HTML forms 

access with Flask 226 

building 213-215 

displaying 218 

producing results 229-230 

redirecting to avoid unwanted errors 234-235 

rendering templates from Flask 216—217 

testing template code 219-221 

html module 11 

HTTP (HyperText Transfer Protocol) 

status codes 222 
web servers and 366 

i 

id built-in function 328 
IDLE (Python IDE) 3-7, 203, 553 
if statement 16-17, 117-119 
ImmutableMultiDict dictionary 261 
ImportError exception 176—177 


import statement 
about 9, 28—29 
Flask framework and 205 
interpreter search considerations 174-177 
positioning 303 
sharing modules witth 173 
threading module and 465 
Zen of Python 567 

increment operator 106,318 
indentation levels for suites 18, 45 

indenting suites of code 
about 15—18, 40 
for functions 147 
for loops 24,27 

index values, lists and 63, 75 
informational messages 222 

_init_method 323-327, 330, 338-340, 443 

inner functions 388, 400 

in operator 
about 15 

dictionaries and 115-119 
lists and 56, 59 
sets and 125 

input built-in function 60 
insert method 65 

INSERT statement (SQL) 289, 463-464 
InterfaceError exception 423, 441, 443 

interpreter (Python) 
about 7—8 

alternative implementations 561 
asking for help 31,41 
case sensitivity 116 
dictionary keys and 108 
functions and 148 
identifying operating system 10 
identifying site-package locations 174 
internal ordering used by 52, 108 
running from command-line 175—177 
syntax errors 5, 57 
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whitespace and 40 

intersection method 125, 128, 159, 167 

ipython shell 552 

IronPython project 561 

isoformat function (datetime module) 11 

items method 110 

itertools module 545 

J 

Java VM 7 

Jinja2 template engine 
about 214-215, 229 
calculating data needed 230 
readable output with 274, 276 

join method 67, 258, 268 
Jones, Brian K. 569 
Jupyter Notebook IDE 554 
Jython project 561 

K 

keyboard shortcuts 2 7 
KeyError exception 115—121 

key/value pairs (dictionaries) 
about 52, 96 
adding 101 

creating on the fly 115—120 
interpreter processing and 108 

keyword assignment of arguments 171 

Kivy library 560 

**kwargs keyword 392—393, 401 

L 

len built-in function 58 

level of indentation for suites 18 

list built-in function 42, 126 

list comprehensions 493, 504, 506, 508-511 


about 13,50-51,54,89 

assignment operators 13, 55, 72-74 

checking for membership in 15, 56, 59 

copying existing 7 2 

creating literally 55 

dictionaries of 485-486 

dynamic 50-51,62 

empty 55, 58, 161 

extending with objects 64 

growing at run-time 58 

iterating over a sequence of objects 24-25 

of arguments 390-391 

popping objects off 63 

reading GSV data as 479 

removing objects from 62 

slice notation 7 7-81, 85 

sorted 126 

spotting in code 54 

spotting pattern with 490 

square bracket notation 13, 54, 66, 74-80, 85 

starting and stopping with 7 8 

steppingwith 79 

tuples and 51, 132 

when not to use 90—91 

workingwith 56,71 

working within edit window 5 

literal lists 54-55 
localhost 211 

logins/logouts 374-381, 384 

logs and logging. See also database-enabling webapps 
determining structure for 288 
dir built-in function and 259-260 
examining raw data 256 
open, process, close technique 250, 253 
single line of delimited data 262 
updating webapps 350-356 
viewing through webapps 254, 258 

loopback address 211 

loop comprehensions. See comprehensions 

loops. See specihc types of loops 
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M 

MacPorts package manager 525 
maps. See dictionaries 
MariaDB 282-283 
Markup object (Flask) 257 
matplodib/seaborn modules 555 
memory management 62 
messages, HTTP status codes 222, 235 

methods (behavior) 
about 49 

arguments and 317, 319-320, 322 
attributes and 322 
chaining method calls 483 
classes and 311-312, 318 
decorators adjusting 207 
functions and 316, 322 
invoking 316—317 
objects and 313, 315, 322 
running webapp 208 

Micro Python project 561 

modules 
about 8 

adding to site-packages 178 
creating 173 
functions and 9,173 
ImportError exception 176—177 
importing 29, 173—174 
sharing code 183 
third party 12 

MongoDB 558 

month attribute (date.today) 11 
multiplication operator 87 
multiprocessing module 546 

MySQL 

benefits of 358 
DB-API and 284 


installing MySQL server 283 
querying considerations 462-463 

MySQL console 287 

N 

NameError exception 321 
namespaces 29 

_name_value 206 

naming conventions 5 
nested functions 388, 400 
newsletters (Python) 566 
Not Found message 208 
not in operator 59, 118—119 
numbers 

assigning to variables 48 
generating randomly 20, 30—31 

numpy package 555 

o 

object class 324 

object instantiation 312,323 

object-oriented programming (OOP) 31 

objects 

about 48—53 

attributes and 313, 315, 322 
classes and 312-313 
creating 312, 323 

dehning representation of 328-329 
duplicate 53, 59 
extending lists with 64 
function 386—389, 395, 397—398 
key/value pairs and 96 
methods and 313, 315, 322 
popping off lists 63 
removing from lists 62 
sequence of 24-25,124 
webapp 206 

open function 245—246 


exception handling and 418, 420, 422, 440, 448—455 
installing MySQL-Gonnector/Python driver 285—286 


, 324, 542 
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opening editing window 3 

open, process, close technique 
about 245 

invoking logging function 250,253 
reading data from existing files 246 
with statement and 247-248 

operating system, identifying for interpreter 10 
operators 

assignment 13, 55, 72-74 

checking for membership with 15, 56, 59, 117—119 

comparison 13, 15 

concatenation 543 

decrement 106 

increment 106, 318 

multiplication 87 

super 15 

ternary 117 

ordered data structures 50-51 
OrderedDict dictionary 545 

os module 
about 9 

environ attribute 10 
getcwd function 9—10 
platform attribute 10 
usage example 10—11 

output display 

exception handling and 255 

Python Shell and 22 

raw data to readable 265-266, 274 

readable via Jinja2 276 

specifying dictionary ordering for 109—110 

P 

pandas tools 555 

parentheses 0 

comprehensions and 506 
function arguments in 149 
object instantiation and 312 
return statement and 158 
tuples in 132, 134 


partial function 545 
pass keyword 312 
pausing execution 20, 28 
pdb debugger 549 
pep8 plug-in 189-190 

PEP (Python Enhancement Protocol) 
about 153 

DB-API specihcation 284 

line length Standard 263 

testing for compliance 188-193, 548 

PermissionError exception 426, 431 
permutations function (itertools module) 545 
Peters, Tim 567 

pip (Package Installer for Python) 
downloading requests library 509 
installing Flask 202 
installing packages with 182 
installing pep8 plug-in 189-190 
installing pyreadline module 524 
installing pytest testing framework 189— 190 

platform attribute (os module) 10 
podcasts (Python) 566 
pop method 63 

positional assignment of arguments 171 
PostgreSQL 282 
POST method (HTTP) 222-223 
pprint function (pprint module) 139 
pprint module 139 

print built-in function 
about 15 

accessing dictionary data values 108 
default behavior 247 
displaying objects 329 
identifying Python version 10 
optional arguments 263 

product function (itertools module) 545 

programming tools 559 

prompt (Python Shell). See Python Shell 
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protocol port number 204, 211 

PSF (Python Software Foundation) 564 

ptpython REPL 553 

PyCharm IDE 553 

py command 175,190 

PyCon (Python Gonference) 564 

PyLint tool 559 

pymongo database driver 558 

PyPI (Python Package Index) 183, 202, 557 

Py Py proj e et 561 

pyreadline module 524 

pytest testing framework 189—190 

py.test tool 548, 559 

Python 2 540 

Python 3 
about 310 

installing on Linux 527 
installing on Mac OS X 525-526 
installing on Windows 522-524 
usage recommendations 540 

PythonAnywhere 
about 529 

conhguring webapps 536 

creating starter webapp 534-535 

extracting and installing code 533 

preparing webapps 530 

signing up for 531 

testing deployed webapp 537 

uploading hies to the cloud 238, 240, 532 

Python community , xxvi—xxx 

Python Core Developers 561 

Python Packaging Authority 183 

Python Shell 
about 4 

accessing prompt within 10, 21-22 
alternatives to 552 
asking for help 31,41 
copying code to editor 57 


experimenting at 21,23-32 
recalling last commands typed 31 
running interpreter from 175—177 
terminating statements with Enter key 24 

Q 

quit command 175 

quotation marks 

comments and 147 
strings and 77, 152 

R 

Ramalho, Luciano 569 

randint function (random module) 20, 30-31, 174 

random module 20, 30-31, 174 

random number generation 20, 30—31 

range built-in function 25, 40—42 

reading 

GSV data as dictionaries 80 
GSV data as lists 479 
data from existing hies 246 

README.txt hle 179-181 
redirect function (Flask) 234-235 
redirection messages 222 
remove method 62 

render_template function (Flask) 217-218, 234 
REPL tool 4, 553 

request object (Flask) 226-227, 259-260, 324 
requests library 509, 557 
requests module 557-558 
results, functions returning 156—159 
return keyword 147 

retura statement 
about 156 
parentheses and 158 
returning multiple values 159 
returning one value 158 
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return values (functions) 
about 156 

interpreter processing 148 
variable scope and 322 

route decorator 
about 209 
adding 217-218 
adjusting behaviors 207 
optional arguments 223 

run-time 

exception handling 115—121, 423—424, 474 
growing dictionaries at 101 
growing lists at 58 

RuntimeError exception 423 

S 

scikit-learn tools 555 

scipy modules 555 

scope of variables 321-322 

SELEGT statement (SQL) 464 

self argument 317, 319-320, 322 

sequence of objects 24-25, 124 

server error messages 222 

session dictionary (flask module) 368—379 

sessions 

about 367 

managing logins/logouts with 374-381 
state and 368-373 

set built-in function 124—125, 160—161, 167 
set comprehensions 504-505 
setdefault method 119-121 
sets 

about 53, 123 
combining 125—126 
commonality between 125, 128 
creating efficiently 124 
creating from sequences 124 
difference between 125,127 
duplicate objects and 53, 59, 123 


empty 160—161 
spotting in code 123 

setup function (setuptools module) 179 
setuptools module 178—179 
single-object tuples 134 
site-packages 174, 177-179 
sleep function (time module) 20, 28 

slice notation 

for loop and 87-88 
lists and 7 7-81, 85 

_slots_directive 542 

sorted built-in function 
about 544 

dictionaries and 109-110 
sets and 123, 126 

spaces versus tabs 40 

split method 268, 270, 479, 482-483 

sqlalchemy module 558 

SQLError exception 453—454 

SQL injection (SQLi) 419, 422 

SQLite 282 

square brackets [ j 

comprehensions and 506 
dictionaries and 99-101, 141 
lists and 13, 54, 66, 74-80, 85 
tuples and 133 

Standard library 
about 9, 10, 146 

additional information 12, 545, 547 
cautions adding/removing modules 178 
concurrency options 465 
identifying locations 174 
usage examples 8, 10—11 

startvalue 41, 76, 78 

state. See attributes (state) 

statements 

assignment 13—14 
control 16 
displaying output 22 
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empty 312 
reusability of 302 
terminating with Enter key 22—24 
try...except 424—431, 434, 441—442 

@staticmethod decorator 542 
status codes (HTTP) 222 
step value 41, 76, 79 
stop value 41, 76, 78 
storing data 

in databases and tables, xvi—xxxviii 
in data structures, xii—xxxviii 
in text files 245 

strftime function (time module) 11 
strings 

assigning to variables 48 
empty 157 
formatting 543 

iterating a specific number of times 24-25 

joining together 67, 258, 268 

key/value pairs and 96 

quotation marks and 77, 152 

splitting apart 268 

turning into list of letters 78 

whitespace and 482 

strip method 482-483 

submodules 8 

success messages 222, 235 

sudo command 190, 202, 527 

suites of code 

comments in 147 
embedded suites within 18 
functions and 147 
indentation levels and 18 
indenting 15-18, 24, 27, 40 
looping 20, 24-27 
running multiple times 20 
unindenting 27 

super operators 15 

syntax errors 5, 57, 312 


sys module 10, 429—430 

T 

tables 288-289, 296. See also dictionaries 
<table> tag 274 
tabs versus spaces 40 
<td> tag 274 

template engines 
about 213—215 

embedding display logic in 275 
preparing to run code 219-221 
relating to web pages 216 
rendering from Flask 217—218 

ternary operator 117 

testing developer tools 189-190, 548 

text files, saving data to 245 

text mode 247 

Thread class 465-466 

threading library 465 

threading module 465, 469—470, 546 

<th> tag 274 

time module 11, 20, 28 

tkinter library 547 

today function (datetime module) 11 

trailing whitespace 40 

<tr> tag 274 

True value 156—157 

try...except statements 424—431, 434, 441—442 
tuples 

about 51, 132—133 
comprehensions and 504, 507 
empty 161 
lists and 51, 132 
single-object 134 
spotting in code 132 

turtle module 547 
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type built-in function 132, 328 
TypeError exception 319, 326-327, 330 
type hints (annotations) 162-163 

u 

unindenting suites of code 27 
union method 125—126 
unittest module 548 
unordered data structures 52—53 

URLs 

function decorators and 207, 209, 211, 218, 223, 396, 
408 

functions with multiple 236 
Processing with generators 510-511 
Processing with list comprehensions 509 
restricting access to 382-383, 396, 408 

v 

van Rossum, Guido 39, 564 
variables 

assigning objects to 48-49 
assigning values to 13, 48-49 
displaying values of 22 
dynamic assignment of 13, 48—49 
environment 10 
functions and 321-322 
global 366 

initializing values 323—325 
misspelling 45 
scope of 321-322 
usage example 13 

venv technology 541 

version, identifying 10 

vertical bar 262 

vim text editor 553 

virtualenv module 541 

virtual programming environments 541 


W 

web applications. See also database-enabling webapps 
adding hnishing touch 234 
adding robustness to 455 
automatic reloading 227 
calculating data needed 230 
creating Flash webapp object 206 
deploying with PythonAnywhere 529—537 
edit/stop/start/test cycle 224-225 
exception handling and 255, 418-420, 422, 433, 
437-439 

exposing functionality to the Web 209—210 

function decorators 207, 209, 211, 218, 223 

functions with multiple URLs 236 

global variables and 366 

handling posted data 223 

how they work 198—199 

HTML forms 213-221,226 

HTTP status codes 222 

installing and using Flask 202-203 

preparing for the cloud 238-240 

producing results as HTML 229-230 

redirecting to avoid unwanted errors 235 

request data in 227 

restarting 210, 220 

running behaviors 208 

running for hrst time 204—205 

stopping 210, 220 

testing 210, 220 

updating 348-349 

viewing logs 254 

web development technologies 556 
what do we want them to do 200, 212 
what happens on the web server 201 

web development technologies 556 

web servers 

about 365—366 

HTTP status codes and 222 

webapp process and 198, 201, 366 

while loop 24 

whitespace 40, 192, 482, 518 
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WingWare IDE 553 
with statement 

classes and 305, 310, 337-338 
context management protocol and 339 
exception handling and 443, 451—452 
open, process, close technique and 247-248 
split method and 482 
viewing logs through webapps 254 

wonder name 206 

wraps funcdon (functools module) 401 


X 

xkcd webcomic 567 

XSS (Gross-site Scripting) 419, 422 

Y 

year attribute (date.today) 11 

z 

Zen of Python 567 
ZIP files 180 
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